2018-03-05 20:17:46.967519 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b0f2ae780 prio 63 cost 48 latency 0.000169 osd_op(client.9244044.0:72 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808) v8 pg pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean]
2018-03-05 20:17:46.967534 7fb2f0710700 20 osd.2 157808 share_map client.9244044 [XXXX::7930]:0/51387864 157808
2018-03-05 20:17:46.967540 7fb2f0710700 20 osd.2 157808 should_share_map client.9244044 [XXXX::7930]:0/51387864 157808
2018-03-05 20:17:46.967544 7fb2f0710700  6 -- op tracker --  seq: 1402, time: 2018-03-05 20:17:46.967543, event: reached_pg, op: osd_op(client.9244044.0:72 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808)
2018-03-05 20:17:46.967550 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] _handle_message: 0x557b0f2ae780
2018-03-05 20:17:46.967563 7fb2f0710700 10 throttle(osd_client_bytes 0x557b05041c00) put 216 (264 -> 48)
2018-03-05 20:17:46.967571 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_op: op osd_op(client.9244044.0:72 38.c 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] snapc 0=[] ondisk+write+known_if_redirected e157808) v8
2018-03-05 20:17:46.967590 7fb2f0710700 10 osd.2 157808 class rgw method user_usage_log_trim flags=rw
2018-03-05 20:17:46.967594 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] op_has_sufficient_caps session=0x557b10de6800 pool=38 (.usage ) owner=0 need_read_cap=0 need_write_cap=0 classes=[class rgw rd 1 wr 1 wl 1] -> yes
2018-03-05 20:17:46.967618 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_op osd_op(client.9244044.0:72 38.c 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] snapc 0=[] ondisk+write+known_if_redirected e157808) v8 may_write may_read -> write-ordered flags ondisk+write+known_if_redirected
2018-03-05 20:17:46.967640 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x557b0ebf59c0
2018-03-05 20:17:46.967650 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] get_object_context: 0x557b0ebf59c0 38:3351fad7:::usage.3:head rwstate(none n=0 w=0) oi: 38:3351fad7:::usage.3:head(157803'741784 osd.2.0:741032 dirty|omap|data_digest|omap_digest s 0 uv 263780 dd ffffffff od 4c08c768 alloc_hint [0 0 0]) exists: 1 ssc: 0x557b10fa7060 snapset: 0=[]:{}
2018-03-05 20:17:46.967668 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] find_object_context 38:3351fad7:::usage.3:head @head oi=38:3351fad7:::usage.3:head(157803'741784 osd.2.0:741032 dirty|omap|data_digest|omap_digest s 0 uv 263780 dd ffffffff od 4c08c768 alloc_hint [0 0 0])
2018-03-05 20:17:46.967685 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_op obc obc(38:3351fad7:::usage.3:head rwstate(excl n=1 w=0))
2018-03-05 20:17:46.967697 7fb2f0710700  6 -- op tracker --  seq: 1402, time: 2018-03-05 20:17:46.967696, event: started, op: osd_op(client.9244044.0:72 38.c 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] snapc 0=[] ondisk+write+known_if_redirected e157808)
2018-03-05 20:17:46.967704 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] execute_ctx 0x557b0d64d800
2018-03-05 20:17:46.967716 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] execute_ctx 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] ov 157803'741784 av 157808'772974 snapc 0=[] snapset 0=[]:{}
2018-03-05 20:17:46.967726 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean]  taking ondisk_read_lock
2018-03-05 20:17:46.967734 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim]
2018-03-05 20:17:46.967742 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op  call rgw.user_usage_log_trim
2018-03-05 20:17:46.967752 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] call method rgw.user_usage_log_trim
2018-03-05 20:17:46.967762 7fb2f0710700 10 <cls> /tmp/release/Ubuntu/WORKDIR/ceph-12.2.2-869-g3570137b77-dho-luminous/src/cls/rgw/cls_rgw.cc:3059: rgw_user_usage_log_trim()
2018-03-05 20:17:46.967764 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op 38:3351fad7:::usage.3:head [stat]
2018-03-05 20:17:46.967772 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op  stat
2018-03-05 20:17:46.967784 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] stat oi has 0 2018-03-01 15:02:42.503138
2018-03-05 20:17:46.967797 7fb2f0710700 10 <cls> /tmp/release/Ubuntu/WORKDIR/ceph-12.2.2-869-g3570137b77-dho-luminous/src/cls/rgw/cls_rgw.cc:2912: usage_iterate_range
2018-03-05 20:17:46.967801 7fb2f0710700 20 <cls> /tmp/release/Ubuntu/WORKDIR/ceph-12.2.2-869-g3570137b77-dho-luminous/src/cls/rgw/cls_rgw.cc:2940: usage_iterate_range start_key=00000000000
2018-03-05 20:17:46.967803 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op 38:3351fad7:::usage.3:head [omap-get-vals]
2018-03-05 20:17:46.967812 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op  omap-get-vals
2018-03-05 20:17:46.967820 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 38.c_head/#38:3351fad7:::usage.3:head#
2018-03-05 20:17:46.967865 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] Found key 01519916400_justin-benjamin_
2018-03-05 20:17:46.967880 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] Found key 01519916400_justin-west-1-staging_
2018-03-05 20:17:46.967895 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] Found key justin-benjamin_01519916400_
2018-03-05 20:17:46.967907 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] Found key justin-west-1-staging_01519916400_
2018-03-05 20:17:46.967928 7fb2f0710700 20 <cls> /tmp/release/Ubuntu/WORKDIR/ceph-12.2.2-869-g3570137b77-dho-luminous/src/cls/rgw/cls_rgw.cc:2960: usage_iterate_range reached key=01519916400_justin-benjamin_, done
2018-03-05 20:17:46.967934 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] method called response length=0
2018-03-05 20:17:46.967945 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean]  dropping ondisk_read_lock
2018-03-05 20:17:46.967958 7fb2f0710700 15 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_osd_op_effects client.9244044 con 0x557b0eb83000
2018-03-05 20:17:46.967968 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] execute_ctx update_log_only -- result=0
2018-03-05 20:17:46.967982 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] record_write_error r=0
2018-03-05 20:17:46.967994 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] submit_log_entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0
2018-03-05 20:17:46.968010 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] new_repop: repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=0 applied?=0 r=0)
2018-03-05 20:17:46.968023 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772973 (157803'740282,157808'772973] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] merge_new_log_entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0
2018-03-05 20:17:46.968035 7fb2f0710700 20 update missing, append 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0
2018-03-05 20:17:46.968046 7fb2f0710700  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 157808'772974, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-03-05 20:17:46.968063 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 lua=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] merge_new_log_entries peer_missing for 14 = missing(0 may_include_deletes = 1)
2018-03-05 20:17:46.968076 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 lua=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] merge_new_log_entries peer_missing for 24 = missing(0 may_include_deletes = 1)
2018-03-05 20:17:46.968091 7fb2f0710700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.14 [2607:f298:4:3245::7106]:6804/17339 existing 0x557b0d538800
2018-03-05 20:17:46.968099 7fb2f0710700 20 osd.2 157808 share_map_peer 0x557b0d538800 already has epoch 157808
2018-03-05 20:17:46.968101 7fb2f0710700  1 -- [2607:f298:4:3245::7108]:6807/48758 --> [2607:f298:4:3245::7106]:6804/17339 -- pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2 -- 0x557b113c5cc0 con 0
2018-03-05 20:17:46.968109 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).prepare_send_message m pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968120 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).prepare_send_message encoding features 2305244844532236283 0x557b113c5cc0 pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968131 7fb2f0710700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).send_message inline write is denied, reschedule m=0x557b113c5cc0
2018-03-05 20:17:46.968139 7fb2f0710700 20 Event(0x557b05026480 nevent=5000 time_id=41).wakeup
2018-03-05 20:17:46.968152 7fb2f0710700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.24 [2607:f298:4:3245::7097]:6800/25906 existing 0x557b0d5a2000
2018-03-05 20:17:46.968159 7fb2f0710700 20 osd.2 157808 share_map_peer 0x557b0d5a2000 already has epoch 157808
2018-03-05 20:17:46.968161 7fb2f0710700  1 -- [2607:f298:4:3245::7108]:6807/48758 --> [2607:f298:4:3245::7097]:6800/25906 -- pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2 -- 0x557b113c5a80 con 0
2018-03-05 20:17:46.968172 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).prepare_send_message m pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968181 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).prepare_send_message encoding features 2305244844532236283 0x557b113c5a80 pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968181 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).handle_write
2018-03-05 20:17:46.968191 7fb2f0710700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).send_message inline write is denied, reschedule m=0x557b113c5a80
2018-03-05 20:17:46.968196 7fb2f0710700 20 Event(0x557b05026480 nevent=5000 time_id=41).wakeup
2018-03-05 20:17:46.968202 7fb2f0710700  5 filestore(/srv/ceph/osd/2) queue_transactions(2160): existing 0x557b050c3e40 osr(38.c 0x557b057ca430)
2018-03-05 20:17:46.968202 7fb308ad1700 10 _calc_signature seq 16 front_crc_ = 4201441152 middle_crc = 0 data_crc = 0 sig = 8778983620244638991
2018-03-05 20:17:46.968205 7fb308ad1700 20 Putting signature in client message(seq # 16): sig = 8778983620244638991
2018-03-05 20:17:46.968205 7fb2f0710700 10 journal prepare_entry [Transaction(0x557b10b2d080)]
2018-03-05 20:17:46.968206 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).write_message signed m=0x557b113c5cc0): sig = 8778983620244638991
2018-03-05 20:17:46.968213 7fb2f0710700 10 journal  len 610 -> 4096 (head 40 pre_pad 0 bl 610 post_pad 3406 tail 40) (bl alignment -1)
2018-03-05 20:17:46.968212 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).write_message sending message type=114 src osd.2 front=192 data=0 off 0
2018-03-05 20:17:46.968218 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).write_message sending 16 0x557b113c5cc0
2018-03-05 20:17:46.968225 7fb2f0710700 10 journal op_submit_start 19665632
2018-03-05 20:17:46.968226 7fb2f0710700  5 filestore(/srv/ceph/osd/2) queue_transactions(2214): (writeahead) 19665632 [Transaction(0x557b10b2d080)]
2018-03-05 20:17:46.968228 7fb2f0710700 10 journal op_journal_transactions 19665632
2018-03-05 20:17:46.968230 7fb2f0710700  5 journal submit_entry seq 19665632 len 4096 (0x557b057cad90)
2018-03-05 20:17:46.968237 7fb2f0710700 10 journal op_submit_finish 19665632
2018-03-05 20:17:46.968237 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0)._try_send sent bytes 267 remaining bytes 0
2018-03-05 20:17:46.968242 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b0f2ae780 finish
2018-03-05 20:17:46.968244 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.968243 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).write_message sending 0x557b113c5cc0 done.
2018-03-05 20:17:46.968247 7fb2f0710700 20 osd.2 op_wq(2) _process empty q, waiting
2018-03-05 20:17:46.968248 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 0
2018-03-05 20:17:46.968244 7fb3013fa700 20 journal write_thread_entry woke up
2018-03-05 20:17:46.968251 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).handle_write
2018-03-05 20:17:46.968254 7fb3013fa700 10 journal room 10238988287 max_size 10239000576 pos 3798675456 header.start 3798667264 top 4096
2018-03-05 20:17:46.968261 7fb3013fa700 10 journal check_for_full at 3798675456 : 4096 < 10238988287
2018-03-05 20:17:46.968263 7fb308ad1700 10 _calc_signature seq 9 front_crc_ = 4201441152 middle_crc = 0 data_crc = 0 sig = 15060623911620606620
2018-03-05 20:17:46.968263 7fb3013fa700 15 journal prepare_single_write 1 will write 3798675456 : seq 19665632 len 610 -> 4096
2018-03-05 20:17:46.968266 7fb308ad1700 20 Putting signature in client message(seq # 9): sig = 15060623911620606620
2018-03-05 20:17:46.968270 7fb3013fa700 20 journal prepare_multi_write queue_pos now 3798679552
2018-03-05 20:17:46.968268 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).write_message signed m=0x557b113c5a80): sig = 15060623911620606620
2018-03-05 20:17:46.968272 7fb3013fa700 15 journal do_aio_write writing 3798675456~4096
2018-03-05 20:17:46.968274 7fb3013fa700 20 journal write_aio_bl 3798675456~4096 seq 19665632
2018-03-05 20:17:46.968273 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).write_message sending message type=114 src osd.2 front=192 data=0 off 0
2018-03-05 20:17:46.968277 7fb3013fa700 20 journal write_aio_bl .. 3798675456~4096 in 1
2018-03-05 20:17:46.968278 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).write_message sending 9 0x557b113c5a80
2018-03-05 20:17:46.968290 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0)._try_send sent bytes 267 remaining bytes 0
2018-03-05 20:17:46.968294 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).write_message sending 0x557b113c5a80 done.
2018-03-05 20:17:46.968299 7fb3013fa700 20 journal write_aio_bl io_submit return value: 1
2018-03-05 20:17:46.968305 7fb3013fa700  5 journal complete_write finished 1 ops and 610 bytes
2018-03-05 20:17:46.968307 7fb3013fa700 20 journal write_thread_entry going to sleep
2018-03-05 20:17:46.968310 7fb300bf9700 20 journal write_finish_thread_entry waiting for aio(s)
2018-03-05 20:17:46.968445 7fb300bf9700 10 journal write_finish_thread_entry aio 3798675456~4096 done
2018-03-05 20:17:46.968451 7fb300bf9700 20 journal check_aio_completion
2018-03-05 20:17:46.968452 7fb300bf9700 20 journal check_aio_completion completed seq 19665632 3798675456~4096
2018-03-05 20:17:46.968458 7fb300bf9700 20 journal check_aio_completion queueing finishers through seq 19665632
2018-03-05 20:17:46.968460 7fb300bf9700 10 journal queue_completions_thru seq 19665632 queueing seq 19665632 0x557b057cad90 lat 0.000227
2018-03-05 20:17:46.968470 7fb300bf9700 20 journal write_finish_thread_entry sleeping
2018-03-05 20:17:46.968482 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968492 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_TAG_ACK pgs=1575 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968497 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_TAG_ACK pgs=1575 cs=1 l=0).process got ACK
2018-03-05 20:17:46.968501 7fb308ad1700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_TAG_ACK pgs=1575 cs=1 l=0).handle_ack got ack seq 16
2018-03-05 20:17:46.968504 7fb3003f8700 10 finisher(0x557b05298020) finisher_thread doing [0x557b057cad90]
2018-03-05 20:17:46.968551 7fb3003f8700  5 filestore(/srv/ceph/osd/2) _journaled_ahead(2308): 0x557b1106c3c0 seq 19665632 osr(38.c 0x557b057ca430) [Transaction(0x557b10b2d080)]
2018-03-05 20:17:46.968506 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_TAG_ACK pgs=1575 cs=1 l=0).handle_ack got ack seq 16 >= 16 on 0x557b113c5cc0 pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968528 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968560 7fb3003f8700  5 filestore(/srv/ceph/osd/2) queue_op(2037): 0x557b1106c3c0 seq 19665632 osr(38.c 0x557b057ca430) 596 bytes   (queue has 1 ops and 596 bytes)
2018-03-05 20:17:46.968559 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_TAG_ACK
2018-03-05 20:17:46.968568 7fb3003f8700 10 filestore(/srv/ceph/osd/2)  queueing ondisk 0x557b10e98860
2018-03-05 20:17:46.968567 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968574 7fb3003f8700 10 finisher(0x557b05298020) finisher_thread done with [0x557b057cad90]
2018-03-05 20:17:46.968577 7fb3003f8700 10 finisher(0x557b05298020) finisher_thread empty
2018-03-05 20:17:46.968574 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_TAG_ACK pgs=4501 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968578 7fb3003f8700 10 finisher(0x557b05298020) finisher_thread sleeping
2018-03-05 20:17:46.968578 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_TAG_ACK pgs=4501 cs=1 l=0).process got ACK
2018-03-05 20:17:46.968585 7fb2febf5700 10 finisher(0x557b050c2580) finisher_thread doing [0x557b10e98860]
2018-03-05 20:17:46.968583 7fb308ad1700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_TAG_ACK pgs=4501 cs=1 l=0).handle_ack got ack seq 9
2018-03-05 20:17:46.968589 7fb2febf5700 10 finisher(0x557b050c2580) finisher_thread done with [0x557b10e98860]
2018-03-05 20:17:46.968578 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=2192 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968590 7fb2febf5700 10 finisher(0x557b050c2580) finisher_thread empty
2018-03-05 20:17:46.968592 7fb2febf5700 10 finisher(0x557b050c2580) finisher_thread sleeping
2018-03-05 20:17:46.968587 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_TAG_ACK pgs=4501 cs=1 l=0).handle_ack got ack seq 9 >= 9 on 0x557b113c5a80 pg_update_log_missing(38.c epoch 157808/157807 rep_tid 1 entries 157808'772974 (0'0) error    38:3351fad7:::usage.3:head by client.9244044.0:72 0.000000 0) v2
2018-03-05 20:17:46.968591 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=2192 cs=1 l=0).process begin MSG
2018-03-05 20:17:46.968597 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=2192 cs=1 l=0).process got MSG header
2018-03-05 20:17:46.968598 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_TAG_ACK
2018-03-05 20:17:46.968602 7fb2ffbf7700 12 FileStore::op_tp worker wq FileStore::OpWQ start processing 0x557b050c3e40 (1 active)
2018-03-05 20:17:46.968602 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=2192 cs=1 l=0).process got envelope type=93 src osd.26 front=192 data=0 off 0
2018-03-05 20:17:46.968606 7fb2ffbf7700 20 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fb2ffbf7700' grace 60 suicide 180
2018-03-05 20:17:46.968607 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:46.968610 7fb2ffbf7700 10 journal op_apply_start 19665632 open_ops 0 -> 1
2018-03-05 20:17:46.968613 7fb2ffbf7700  5 filestore(/srv/ceph/osd/2) _do_op(2080): 0x557b1106c3c0 seq 19665632 osr(38.c 0x557b057ca430)/0x557b057ca430 start
2018-03-05 20:17:46.968612 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:46.968615 7fb2ffbf7700 10 filestore(/srv/ceph/osd/2) _do_transaction(2621): on 0x557b10b2d080
2018-03-05 20:17:46.968616 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:46.968620 7fb3092d2700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) get_or_fail 192 success (0 -> 192)
2018-03-05 20:17:46.968625 7fb2ffbf7700 20 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fb2ffbf7700' grace 60 suicide 180
2018-03-05 20:17:46.968623 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:46.968629 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=2192 cs=1 l=0).process got front 192
2018-03-05 20:17:46.968632 7fb2ffbf7700 15 filestore(/srv/ceph/osd/2) _omap_setkeys(5508): 38.c_head/#38:30000000::::head#
2018-03-05 20:17:46.968635 7fb2ffbf7700 20 filestore(/srv/ceph/osd/2) _omap_setkeys(5532):  set 0000157808.00000000000000772974
2018-03-05 20:17:46.968633 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2192 cs=1 l=0).process aborted = 0
2018-03-05 20:17:46.968636 7fb2ffbf7700 20 filestore(/srv/ceph/osd/2) _omap_setkeys(5532):  set _fastinfo
2018-03-05 20:17:46.968637 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2192 cs=1 l=0).process got 192 + 0 + 0 byte message
2018-03-05 20:17:46.968643 7fb2ffbf7700 10 filestore oid: #38:30000000::::head# not skipping op, *spos 19665632.0.0
2018-03-05 20:17:46.968646 7fb2ffbf7700 10 filestore  > header.spos 0.0.0
2018-03-05 20:17:46.968665 7fb3092d2700 10 _calc_signature seq 1216 front_crc_ = 2495887566 middle_crc = 0 data_crc = 0 sig = 863811742372077520
2018-03-05 20:17:46.968675 7fb2ffbf7700 20 filestore(/srv/ceph/osd/2) _omap_setkeys(5536): 38.c_head/#38:30000000::::head# = 0
2018-03-05 20:17:46.968668 7fb3092d2700  5 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=2192 cs=1 l=0). rx osd.26 seq 1216 0x557b0eb76800 replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7) v7
2018-03-05 20:17:46.968679 7fb2ffbf7700 20 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fb2ffbf7700' grace 60 suicide 180
2018-03-05 20:17:46.968681 7fb2ffbf7700 10 journal op_apply_finish 19665632 open_ops 1 -> 0, max_applied_seq 19665631 -> 19665632
2018-03-05 20:17:46.968683 7fb2ffbf7700 10 filestore(/srv/ceph/osd/2) _do_op(2085): 0x557b1106c3c0 seq 19665632 r = 0, finisher 0x557b05475510 0
2018-03-05 20:17:46.968681 7fb3092d2700  1 -- [2607:f298:4:3245::7108]:6807/48758 <== osd.26 [2607:f298:4:3245::7097]:6804/64507 1216 ==== replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7) v7 ==== 192+0+0 (2495887566 0 0) 0x557b0eb76800 con 0x557b0d537000
2018-03-05 20:17:46.968695 7fb2ffbf7700 10 filestore(/srv/ceph/osd/2) _finish_op(2100): 0x557b1106c3c0 seq 19665632 osr(38.c 0x557b057ca430)/0x557b057ca430 lat 0.000489
2018-03-05 20:17:46.968694 7fb3092d2700 15 osd.2 157808 enqueue_op 0x557b076e1000 prio 127 cost 0 latency 0.000102 epoch 157808 replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7) v7
2018-03-05 20:17:46.968704 7fb2ffbf7700 15 FileStore::op_tp worker wq FileStore::OpWQ done processing 0x557b050c3e40 (0 active)
2018-03-05 20:17:46.968706 7fb2ffbf7700 20 FileStore::op_tp worker waiting
2018-03-05 20:17:46.968707 7fb2ffbf7700 20 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fb2ffbf7700' grace 60 suicide 0
2018-03-05 20:17:46.968707 7fb2fe3f4700 10 finisher(0x557b050c2420) finisher_thread doing [0x557b05475510]
2018-03-05 20:17:46.968701 7fb3092d2700  6 -- op tracker --  seq: 1403, time: 2018-03-05 20:17:46.968701, event: queued_for_pg, op: replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7)
2018-03-05 20:17:46.968712 7fb3092d2700 20 osd.2 op_wq(2) _enqueue 44.7 PGQueueable(0x557b076e1000 prio 127 cost 0 e157808)
2018-03-05 20:17:46.968720 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 dispatch_throttle_release 192 to dispatch throttler 192/104857600
2018-03-05 20:17:46.968724 7fb3092d2700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) put 192 (192 -> 0)
2018-03-05 20:17:46.968712 7fb2fe3f4700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 lua=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] op_applied version 157808'772974
2018-03-05 20:17:46.968726 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 done calling dispatch on 0x557b0eb76800
2018-03-05 20:17:46.968726 7fb2f0710700 20 osd.2 op_wq(2) _process 44.7 item PGQueueable(0x557b076e1000 prio 127 cost 0 e157808) queued
2018-03-05 20:17:46.968730 7fb2fe3f4700 10 finisher(0x557b050c2420) finisher_thread done with [0x557b05475510]
2018-03-05 20:17:46.968732 7fb2fe3f4700 10 finisher(0x557b050c2420) finisher_thread empty
2018-03-05 20:17:46.968731 7fb2f0710700 20 osd.2 op_wq(2) _process 44.7 item PGQueueable(0x557b076e1000 prio 127 cost 0 e157808) pg 0x557b0cc64000
2018-03-05 20:17:46.968730 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:46.968734 7fb2fe3f4700 10 finisher(0x557b050c2420) finisher_thread sleeping
2018-03-05 20:17:46.968738 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).handle_write
2018-03-05 20:17:46.968744 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).handle_write try send msg ack, acked 1 messages
2018-03-05 20:17:46.968735 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b076e1000 prio 127 cost 0 latency 0.000143 replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7) v7 pg pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active]
2018-03-05 20:17:46.968751 7fb2f0710700 20 osd.2 157808 share_map osd.26 [2607:f298:4:3245::7097]:6804/64507 157808
2018-03-05 20:17:46.968755 7fb2f0710700 20 osd.2 157808 should_share_map osd.26 [2607:f298:4:3245::7097]:6804/64507 157808
2018-03-05 20:17:46.968757 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0)._try_send sent bytes 9 remaining bytes 0
2018-03-05 20:17:46.968760 7fb2f0710700  6 -- op tracker --  seq: 1403, time: 2018-03-05 20:17:46.968759, event: reached_pg, op: replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7)
2018-03-05 20:17:46.968766 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] _handle_message: 0x557b076e1000
2018-03-05 20:17:46.968775 7fb2f0710700  7 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] replica_scrub
2018-03-05 20:17:46.968783 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] build_scrub_map_chunk [44:e1488cb6:::.dir.default.6599779.41685.23:0,44:e148c9ca:::.dir.default.6614891.58717.17:0)  seed 4294967295
2018-03-05 20:17:46.968793 7fb2f0710700 20 filestore(/srv/ceph/osd/2) collection_list(4989): pool is 44 shard is 255 pgid 44.7
2018-03-05 20:17:46.968795 7fb2f0710700 10 filestore(/srv/ceph/osd/2) collection_list(5008): start #44:e1488cb6:::.dir.default.6599779.41685.23:0# >= sep #-1:00000000::::0#
2018-03-05 20:17:46.968799 7fb2f0710700 20 _collection_list_partial start:#44:e1488cb6:::.dir.default.6599779.41685.23:0# end:#44:e148c9ca:::.dir.default.6614891.58717.17:0#-2147483647 ls.size 0
2018-03-05 20:17:46.968816 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968830 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=1575 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.968837 7fb2f0710700 20 list_by_hash_bitwise prefix 7
2018-03-05 20:17:46.968835 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=1575 cs=1 l=0).process begin MSG
2018-03-05 20:17:46.968840 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=1575 cs=1 l=0).process got MSG header
2018-03-05 20:17:46.968845 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=1575 cs=1 l=0).process got envelope type=115 src osd.14 front=41 data=0 off 0
2018-03-05 20:17:46.968851 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:46.968855 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:46.968858 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:46.968863 7fb308ad1700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) get_or_fail 41 success (0 -> 41)
2018-03-05 20:17:46.968865 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:46.968870 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=1575 cs=1 l=0).process got front 41
2018-03-05 20:17:46.968877 7fb2f0710700 20 list_by_hash_bitwise prefix 78
2018-03-05 20:17:46.968874 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1575 cs=1 l=0).process aborted = 0
2018-03-05 20:17:46.968879 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1575 cs=1 l=0).process got 41 + 0 + 0 byte message
2018-03-05 20:17:46.968899 7fb308ad1700 10 _calc_signature seq 16 front_crc_ = 4263875987 middle_crc = 0 data_crc = 0 sig = 9518418682615378741
2018-03-05 20:17:46.968901 7fb308ad1700  5 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1575 cs=1 l=0). rx osd.14 seq 16 0x557b100a6c00 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2
2018-03-05 20:17:46.968909 7fb308ad1700  1 -- [2607:f298:4:3245::7108]:6807/48758 <== osd.14 [2607:f298:4:3245::7106]:6804/17339 16 ==== pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2 ==== 41+0+0 (4263875987 0 0) 0x557b100a6c00 con 0x557b0d538800
2018-03-05 20:17:46.968917 7fb308ad1700 15 osd.2 157808 enqueue_op 0x557b0f475980 prio 196 cost 0 latency 0.000081 epoch 157808 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2
2018-03-05 20:17:46.968921 7fb308ad1700  6 -- op tracker --  seq: 1404, time: 2018-03-05 20:17:46.968920, event: queued_for_pg, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:46.968928 7fb308ad1700 20 osd.2 op_wq(2) _enqueue 38.c PGQueueable(0x557b0f475980 prio 196 cost 0 e157808)
2018-03-05 20:17:46.968933 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 dispatch_throttle_release 41 to dispatch throttler 41/104857600
2018-03-05 20:17:46.968936 7fb2f0710700 20 list_by_hash_bitwise prefix 782
2018-03-05 20:17:46.968936 7fb308ad1700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) put 41 (41 -> 0)
2018-03-05 20:17:46.968939 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 done calling dispatch on 0x557b100a6c00
2018-03-05 20:17:46.968941 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:46.968947 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).handle_write
2018-03-05 20:17:46.968952 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0).handle_write try send msg ack, acked 1 messages
2018-03-05 20:17:46.968964 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7106]:6804/17339 conn(0x557b0d538800 :-1 s=STATE_OPEN pgs=1575 cs=1 l=0)._try_send sent bytes 9 remaining bytes 0
2018-03-05 20:17:46.968991 7fb2f0710700 20 list_by_hash_bitwise prefix 7821
2018-03-05 20:17:46.969634 7fb2f0710700 20 list_by_hash_bitwise prefix 782113D6
2018-03-05 20:17:46.969640 7fb2f0710700 20 list_by_hash_bitwise prefix 782113D6 ob #44:e1488cb6:::.dir.default.6599779.41685.23:head#
2018-03-05 20:17:46.969643 7fb2f0710700 20 list_by_hash_bitwise prefix 78211F78
2018-03-05 20:17:46.969644 7fb2f0710700 20 list_by_hash_bitwise prefix 78211F78 ob #44:e1488fe1:::.dir.default.6614891.25332.6:head#
2018-03-05 20:17:46.969646 7fb2f0710700 20 list_by_hash_bitwise prefix 78211FF8
2018-03-05 20:17:46.969647 7fb2f0710700 20 list_by_hash_bitwise prefix 78211FF8 ob #44:e1488ff1:::.dir.default.6599779.21977.25:head#
2018-03-05 20:17:46.969653 7fb2f0710700 20 list_by_hash_bitwise prefix 782194F4
2018-03-05 20:17:46.969654 7fb2f0710700 20 list_by_hash_bitwise prefix 782194F4 ob #44:e14892f2:::.dir.default.6614891.41678.28:head#
2018-03-05 20:17:46.969656 7fb2f0710700 20 list_by_hash_bitwise prefix 78219C5D
2018-03-05 20:17:46.969657 7fb2f0710700 20 list_by_hash_bitwise prefix 78219C5D ob #44:e14893ab:::.dir.default.6614891.51167.1:head#
2018-03-05 20:17:46.969659 7fb2f0710700 20 list_by_hash_bitwise prefix 78219AF2
2018-03-05 20:17:46.969660 7fb2f0710700 20 list_by_hash_bitwise prefix 78219AF2 ob #44:e14895f4:::.dir.default.6582062.5213.28:head#
2018-03-05 20:17:46.969666 7fb2f0710700 20 list_by_hash_bitwise prefix 78219655
2018-03-05 20:17:46.969668 7fb2f0710700 20 list_by_hash_bitwise prefix 78219655 ob #44:e14896aa:::.dir.default.6599779.35066.27:head#
2018-03-05 20:17:46.969669 7fb2f0710700 20 list_by_hash_bitwise prefix 78219D41
2018-03-05 20:17:46.969671 7fb2f0710700 20 list_by_hash_bitwise prefix 78219D41 ob #44:e1489b28:::.dir.default.6586352.19527.6:head#
2018-03-05 20:17:46.969673 7fb2f0710700 20 list_by_hash_bitwise prefix 78219D4D
2018-03-05 20:17:46.969674 7fb2f0710700 20 list_by_hash_bitwise prefix 78219D4D ob #44:e1489b2b:::.dir.default.6614891.24114.16:head#
2018-03-05 20:17:46.969676 7fb2f0710700 20 list_by_hash_bitwise prefix 78219DB2
2018-03-05 20:17:46.969677 7fb2f0710700 20 list_by_hash_bitwise prefix 78219DB2 ob #44:e1489bd4:::.dir.default.6614891.18413.9:head#
2018-03-05 20:17:46.969679 7fb2f0710700 20 list_by_hash_bitwise prefix 78219FEC
2018-03-05 20:17:46.969680 7fb2f0710700 20 list_by_hash_bitwise prefix 78219FEC ob #44:e1489f73:::.dir.default.6708021.15007.27:head#
2018-03-05 20:17:46.969682 7fb2f0710700 20 list_by_hash_bitwise prefix 78215227
2018-03-05 20:17:46.969683 7fb2f0710700 20 list_by_hash_bitwise prefix 78215227 ob #44:e148a44e:::.dir.default.6614891.3366.14:head#
2018-03-05 20:17:46.969685 7fb2f0710700 20 list_by_hash_bitwise prefix 7821566F
2018-03-05 20:17:46.969686 7fb2f0710700 20 list_by_hash_bitwise prefix 7821566F ob #44:e148a66f:::.dir.default.6586352.11407.1:head#
2018-03-05 20:17:46.969688 7fb2f0710700 20 list_by_hash_bitwise prefix 7821552D
2018-03-05 20:17:46.969689 7fb2f0710700 20 list_by_hash_bitwise prefix 7821552D ob #44:e148aa4b:::.dir.default.6614891.37088.27:head#
2018-03-05 20:17:46.969691 7fb2f0710700 20 list_by_hash_bitwise prefix 78215D0D
2018-03-05 20:17:46.969692 7fb2f0710700 20 list_by_hash_bitwise prefix 78215D0D ob #44:e148ab0b:::.dir.default.6599779.39400.1:head#
2018-03-05 20:17:46.969694 7fb2f0710700 20 list_by_hash_bitwise prefix 78215768
2018-03-05 20:17:46.969695 7fb2f0710700 20 list_by_hash_bitwise prefix 78215768 ob #44:e148ae61:::.dir.default.6599779.14040.6:head#
2018-03-05 20:17:46.969696 7fb2f0710700 20 list_by_hash_bitwise prefix 78215F47
2018-03-05 20:17:46.969698 7fb2f0710700 20 list_by_hash_bitwise prefix 78215F47 ob #44:e148af2e:::.dir.default.6586352.39617.10:head#
2018-03-05 20:17:46.969700 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D040
2018-03-05 20:17:46.969701 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D040 ob #44:e148b020:::.dir.default.6599779.15359.29:head#
2018-03-05 20:17:46.969703 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D8B2
2018-03-05 20:17:46.969704 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D8B2 ob #44:e148b1d4:::.dir.default.6582062.4652.23:head#
2018-03-05 20:17:46.969705 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D48E
2018-03-05 20:17:46.969706 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D48E ob #44:e148b217:::.dir.default.6708018.1204.8:head#
2018-03-05 20:17:46.969708 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D6FA
2018-03-05 20:17:46.969709 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D6FA ob #44:e148b6f5:::.dir.default.6586352.27495.8:head#
2018-03-05 20:17:46.969711 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D9C0
2018-03-05 20:17:46.969712 7fb2f0710700 20 list_by_hash_bitwise prefix 7821D9C0 ob #44:e148b930:::.dir.default.6708018.12222.28:head#
2018-03-05 20:17:46.969713 7fb2f0710700 20 list_by_hash_bitwise prefix 7821DDC3
2018-03-05 20:17:46.969714 7fb2f0710700 20 list_by_hash_bitwise prefix 7821DDC3 ob #44:e148bb3c:::.dir.default.6614891.22856.5:head#
2018-03-05 20:17:46.969716 7fb2f0710700 20 list_by_hash_bitwise prefix 7821DB41
2018-03-05 20:17:46.969717 7fb2f0710700 20 list_by_hash_bitwise prefix 7821DB41 ob #44:e148bd28:::.dir.default.6582062.4462.0:head#
2018-03-05 20:17:46.969719 7fb2f0710700 20 list_by_hash_bitwise prefix 78213653
2018-03-05 20:17:46.969720 7fb2f0710700 20 list_by_hash_bitwise prefix 78213653 ob #44:e148c6ac:::.dir.default.6599779.26330.4:head#
2018-03-05 20:17:46.969722 7fb2f0710700 20 list_by_hash_bitwise prefix 78213935
2018-03-05 20:17:46.969732 7fb2f0710700 20 filestore(/srv/ceph/osd/2) objects: [#44:e1488cb6:::.dir.default.6599779.41685.23:head#,#44:e1488fe1:::.dir.default.6614891.25332.6:head#,#44:e1488ff1:::.dir.default.6599779.21977.25:head#,#44:e14892f2:::.dir.default.6614891.41678.28:head#,#44:e14893ab:::.dir.default.6614891.51167.1:head#,#44:e14895f4:::.dir.default.6582062.5213.28:head#,#44:e14896aa:::.dir.default.6599779.35066.27:head#,#44:e1489b28:::.dir.default.6586352.19527.6:head#,#44:e1489b2b:::.dir.default.6614891.24114.16:head#,#44:e1489bd4:::.dir.default.6614891.18413.9:head#,#44:e1489f73:::.dir.default.6708021.15007.27:head#,#44:e148a44e:::.dir.default.6614891.3366.14:head#,#44:e148a66f:::.dir.default.6586352.11407.1:head#,#44:e148aa4b:::.dir.default.6614891.37088.27:head#,#44:e148ab0b:::.dir.default.6599779.39400.1:head#,#44:e148ae61:::.dir.default.6599779.14040.6:head#,#44:e148af2e:::.dir.default.6586352.39617.10:head#,#44:e148b020:::.dir.default.6599779.15359.29:head#,#44:e148b1d4:::.dir.default.6582062.4652.23:head#,#44:e148b217:::.dir.default.6708018.1204.8:head#,#44:e148b6f5:::.dir.default.6586352.27495.8:head#,#44:e148b930:::.dir.default.6708018.12222.28:head#,#44:e148bb3c:::.dir.default.6614891.22856.5:head#,#44:e148bd28:::.dir.default.6582062.4462.0:head#,#44:e148c6ac:::.dir.default.6599779.26330.4:head#]
2018-03-05 20:17:46.969755 7fb2f0710700 20 filestore(/srv/ceph/osd/2)   next #44:e148c9ca:::.dir.default.6614891.58717.17:head#
2018-03-05 20:17:46.969759 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_scan_list scanning 25 objects deeply
2018-03-05 20:17:46.969769 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.969792 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head# = 0 (size 0)
2018-03-05 20:17:46.969796 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head#
2018-03-05 20:17:46.973988 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting '_'
2018-03-05 20:17:46.974007 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting 'snapset'
2018-03-05 20:17:46.974013 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:46.974015 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head# = 0
2018-03-05 20:17:46.974019 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1488cb6:::.dir.default.6599779.41685.23:head seed ffffffff
2018-03-05 20:17:46.974031 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.974033 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head# 0~524288
2018-03-05 20:17:46.974079 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head# 0~0/524288
2018-03-05 20:17:46.974085 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head#
2018-03-05 20:17:46.974143 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1488cb6:::.dir.default.6599779.41685.23:head#
2018-03-05 20:17:46.974179 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1488cb6:::.dir.default.6599779.41685.23:head omap_digest f5109350
2018-03-05 20:17:46.974195 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.974215 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head# = 0 (size 0)
2018-03-05 20:17:46.974221 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head#
2018-03-05 20:17:46.977444 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting '_'
2018-03-05 20:17:46.977462 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting 'snapset'
2018-03-05 20:17:46.977467 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:46.977469 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head# = 0
2018-03-05 20:17:46.977473 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1488fe1:::.dir.default.6614891.25332.6:head seed ffffffff
2018-03-05 20:17:46.977487 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.977489 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head# 0~524288
2018-03-05 20:17:46.977528 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head# 0~0/524288
2018-03-05 20:17:46.977536 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head#
2018-03-05 20:17:46.977592 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1488fe1:::.dir.default.6614891.25332.6:head#
2018-03-05 20:17:46.977628 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1488fe1:::.dir.default.6614891.25332.6:head omap_digest f5109350
2018-03-05 20:17:46.977644 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.977663 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head# = 0 (size 0)
2018-03-05 20:17:46.977670 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head#
2018-03-05 20:17:46.979958 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.979992 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=4501 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:46.979997 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=4501 cs=1 l=0).process begin MSG
2018-03-05 20:17:46.980001 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=4501 cs=1 l=0).process got MSG header
2018-03-05 20:17:46.980006 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=4501 cs=1 l=0).process got envelope type=115 src osd.24 front=41 data=0 off 0
2018-03-05 20:17:46.980011 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:46.980015 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:46.980019 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:46.980024 7fb308ad1700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) get_or_fail 41 success (0 -> 41)
2018-03-05 20:17:46.980026 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:46.980033 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=4501 cs=1 l=0).process got front 41
2018-03-05 20:17:46.980038 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4501 cs=1 l=0).process aborted = 0
2018-03-05 20:17:46.980042 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4501 cs=1 l=0).process got 41 + 0 + 0 byte message
2018-03-05 20:17:46.980070 7fb308ad1700 10 _calc_signature seq 8 front_crc_ = 4263875987 middle_crc = 0 data_crc = 0 sig = 18280068126841891507
2018-03-05 20:17:46.980073 7fb308ad1700  5 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=4501 cs=1 l=0). rx osd.24 seq 8 0x557b100a6a00 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2
2018-03-05 20:17:46.980080 7fb308ad1700  1 -- [2607:f298:4:3245::7108]:6807/48758 <== osd.24 [2607:f298:4:3245::7097]:6800/25906 8 ==== pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2 ==== 41+0+0 (4263875987 0 0) 0x557b100a6a00 con 0x557b0d5a2000
2018-03-05 20:17:46.980090 7fb308ad1700 15 osd.2 157808 enqueue_op 0x557b105a3380 prio 196 cost 0 latency 0.000092 epoch 157808 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2
2018-03-05 20:17:46.980098 7fb308ad1700  6 -- op tracker --  seq: 1405, time: 2018-03-05 20:17:46.980097, event: queued_for_pg, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:46.980105 7fb308ad1700 20 osd.2 op_wq(2) _enqueue 38.c PGQueueable(0x557b105a3380 prio 196 cost 0 e157808)
2018-03-05 20:17:46.980110 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 dispatch_throttle_release 41 to dispatch throttler 41/104857600
2018-03-05 20:17:46.980113 7fb308ad1700 10 throttle(msgr_dispatch_throttler-cluster 0x557b052b71f8) put 41 (41 -> 0)
2018-03-05 20:17:46.980115 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 done calling dispatch on 0x557b100a6a00
2018-03-05 20:17:46.980117 7fb308ad1700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:46.980123 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).handle_write
2018-03-05 20:17:46.980129 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0).handle_write try send msg ack, acked 1 messages
2018-03-05 20:17:46.980142 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6800/25906 conn(0x557b0d5a2000 :-1 s=STATE_OPEN pgs=4501 cs=1 l=0)._try_send sent bytes 9 remaining bytes 0
2018-03-05 20:17:46.984471 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting '_'
2018-03-05 20:17:46.984490 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting 'snapset'
2018-03-05 20:17:46.984495 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:46.984497 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head# = 0
2018-03-05 20:17:46.984501 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1488ff1:::.dir.default.6599779.21977.25:head seed ffffffff
2018-03-05 20:17:46.984514 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.984516 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head# 0~524288
2018-03-05 20:17:46.984558 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head# 0~0/524288
2018-03-05 20:17:46.984563 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head#
2018-03-05 20:17:46.984622 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1488ff1:::.dir.default.6599779.21977.25:head#
2018-03-05 20:17:46.984657 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1488ff1:::.dir.default.6599779.21977.25:head omap_digest f5109350
2018-03-05 20:17:46.984671 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.984691 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head# = 0 (size 0)
2018-03-05 20:17:46.984697 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head#
2018-03-05 20:17:46.988455 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting '_'
2018-03-05 20:17:46.988472 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting 'snapset'
2018-03-05 20:17:46.988477 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:46.988478 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head# = 0
2018-03-05 20:17:46.988482 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e14892f2:::.dir.default.6614891.41678.28:head seed ffffffff
2018-03-05 20:17:46.988494 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.988495 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head# 0~524288
2018-03-05 20:17:46.988531 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head# 0~0/524288
2018-03-05 20:17:46.988541 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head#
2018-03-05 20:17:46.988596 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e14892f2:::.dir.default.6614891.41678.28:head#
2018-03-05 20:17:46.988630 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e14892f2:::.dir.default.6614891.41678.28:head omap_digest f5109350
2018-03-05 20:17:46.988645 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.988664 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head# = 0 (size 0)
2018-03-05 20:17:46.988670 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head#
2018-03-05 20:17:46.992188 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:46.992203 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:46.992231 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=57 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:46.992236 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=57 cs=1 l=1).process begin MSG
2018-03-05 20:17:46.992239 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=59 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:46.992241 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=57 cs=1 l=1).process got MSG header
2018-03-05 20:17:46.992244 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=59 cs=1 l=1).process begin MSG
2018-03-05 20:17:46.992245 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=57 cs=1 l=1).process got envelope type=70 src osd.32 front=2004 data=0 off 0
2018-03-05 20:17:46.992249 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=59 cs=1 l=1).process got MSG header
2018-03-05 20:17:46.992251 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=57 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:46.992254 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=59 cs=1 l=1).process got envelope type=70 src osd.32 front=2004 data=0 off 0
2018-03-05 20:17:46.992255 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=57 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:46.992260 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=59 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:46.992259 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=57 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:46.992265 7fb308ad1700 10 throttle(msgr_dispatch_throttler-hb_front_server 0x557b052b78f8) get_or_fail 2004 success (0 -> 2004)
2018-03-05 20:17:46.992264 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=59 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:46.992268 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=57 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:46.992268 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=59 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:46.992275 7fb3092d2700 10 throttle(msgr_dispatch_throttler-hb_back_server 0x557b052b7ff8) get_or_fail 2004 success (0 -> 2004)
2018-03-05 20:17:46.992274 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=57 cs=1 l=1).process got front 2004
2018-03-05 20:17:46.992279 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=57 cs=1 l=1).process aborted = 0
2018-03-05 20:17:46.992278 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=59 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:46.992283 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=57 cs=1 l=1).process got 2004 + 0 + 0 byte message
2018-03-05 20:17:46.992286 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=59 cs=1 l=1).process got front 2004
2018-03-05 20:17:46.992291 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=59 cs=1 l=1).process aborted = 0
2018-03-05 20:17:46.992295 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=59 cs=1 l=1).process got 2004 + 0 + 0 byte message
2018-03-05 20:17:46.992292 7fb308ad1700  5 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=57 cs=1 l=1). rx osd.32 seq 86 0x557b100a5400 osd_ping(ping e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992301 7fb308ad1700  1 -- [XXXX::7108]:6813/48758 <== osd.32 [XXXX::7097]:0/60421 86 ==== osd_ping(ping e157808 stamp 2018-03-05 20:17:46.992052) v4 ==== 2004+0+0 (3441688654 0 0) 0x557b100a5400 con 0x557b0563d000
2018-03-05 20:17:46.992310 7fb308ad1700 20 heartbeat_map is_healthy = healthy, total workers: 11, number of unhealthy: 0
2018-03-05 20:17:46.992304 7fb3092d2700  5 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=59 cs=1 l=1). rx osd.32 seq 86 0x557b10de4000 osd_ping(ping e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992312 7fb308ad1700  1 -- [XXXX::7108]:6813/48758 --> [XXXX::7097]:0/60421 -- osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4 -- 0x557b100a4800 con 0
2018-03-05 20:17:46.992313 7fb3092d2700  1 -- [2607:f298:4:3245::7108]:6814/48758 <== osd.32 [2607:f298:4:3245::7097]:0/60421 86 ==== osd_ping(ping e157808 stamp 2018-03-05 20:17:46.992052) v4 ==== 2004+0+0 (3441688654 0 0) 0x557b10de4000 con 0x557b0563b800
2018-03-05 20:17:46.992318 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).prepare_send_message m osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992324 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).prepare_send_message encoding features 2305244844532236283 0x557b100a4800 osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992334 7fb308ad1700 15 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).send_message inline write is denied, reschedule m=0x557b100a4800
2018-03-05 20:17:46.992341 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.32 [2607:f298:4:3245::7097]:6806/60421 existing 0x557b0d597000
2018-03-05 20:17:46.992347 7fb308ad1700 20 osd.2 157808 share_map_peer 0x557b0d597000 already has epoch 157808
2018-03-05 20:17:46.992354 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 dispatch_throttle_release 2004 to dispatch throttler 2004/104857600
2018-03-05 20:17:46.992368 7fb308ad1700 10 throttle(msgr_dispatch_throttler-hb_front_server 0x557b052b78f8) put 2004 (2004 -> 0)
2018-03-05 20:17:46.992370 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 done calling dispatch on 0x557b100a5400
2018-03-05 20:17:46.992369 7fb3092d2700 20 heartbeat_map is_healthy = healthy, total workers: 11, number of unhealthy: 0
2018-03-05 20:17:46.992373 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:46.992379 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).handle_write
2018-03-05 20:17:46.992376 7fb3092d2700  1 -- [2607:f298:4:3245::7108]:6814/48758 --> [2607:f298:4:3245::7097]:0/60421 -- osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4 -- 0x557b10de5400 con 0
2018-03-05 20:17:46.992385 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).write_message signed m=0x557b100a4800): sig = 0
2018-03-05 20:17:46.992386 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).prepare_send_message m osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992390 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).write_message sending message type=70 src osd.2 front=2004 data=0 off 0
2018-03-05 20:17:46.992396 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).write_message sending 86 0x557b100a4800
2018-03-05 20:17:46.992394 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).prepare_send_message encoding features 2305244844532236283 0x557b10de5400 osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:46.992052) v4
2018-03-05 20:17:46.992408 7fb3092d2700 15 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).send_message inline write is denied, reschedule m=0x557b10de5400
2018-03-05 20:17:46.992410 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1)._try_send sent bytes 2079 remaining bytes 0
2018-03-05 20:17:46.992416 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7097]:0/60421 conn(0x557b0563d000 :6813 s=STATE_OPEN pgs=57 cs=1 l=1).write_message sending 0x557b100a4800 done.
2018-03-05 20:17:46.992416 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.32 [2607:f298:4:3245::7097]:6806/60421 existing 0x557b0d597000
2018-03-05 20:17:46.992422 7fb3092d2700 20 osd.2 157808 share_map_peer 0x557b0d597000 already has epoch 157808
2018-03-05 20:17:46.992426 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6814/48758 dispatch_throttle_release 2004 to dispatch throttler 2004/104857600
2018-03-05 20:17:46.992429 7fb3092d2700 10 throttle(msgr_dispatch_throttler-hb_back_server 0x557b052b7ff8) put 2004 (2004 -> 0)
2018-03-05 20:17:46.992431 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 done calling dispatch on 0x557b10de4000
2018-03-05 20:17:46.992434 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:46.992443 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).handle_write
2018-03-05 20:17:46.992449 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).write_message signed m=0x557b10de5400): sig = 0
2018-03-05 20:17:46.992454 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).write_message sending message type=70 src osd.2 front=2004 data=0 off 0
2018-03-05 20:17:46.992459 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).write_message sending 86 0x557b10de5400
2018-03-05 20:17:46.992479 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1)._try_send sent bytes 2079 remaining bytes 0
2018-03-05 20:17:46.992485 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7097]:0/60421 conn(0x557b0563b800 :6814 s=STATE_OPEN pgs=59 cs=1 l=1).write_message sending 0x557b10de5400 done.
2018-03-05 20:17:46.994216 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting '_'
2018-03-05 20:17:46.994236 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting 'snapset'
2018-03-05 20:17:46.994242 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:46.994243 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head# = 0
2018-03-05 20:17:46.994248 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e14893ab:::.dir.default.6614891.51167.1:head seed ffffffff
2018-03-05 20:17:46.994261 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.994263 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head# 0~524288
2018-03-05 20:17:46.994309 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head# 0~0/524288
2018-03-05 20:17:46.994316 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head#
2018-03-05 20:17:46.994378 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e14893ab:::.dir.default.6614891.51167.1:head#
2018-03-05 20:17:46.994413 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e14893ab:::.dir.default.6614891.51167.1:head omap_digest f5109350
2018-03-05 20:17:46.994428 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:46.994447 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head# = 0 (size 0)
2018-03-05 20:17:46.994454 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head#
2018-03-05 20:17:47.013702 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting '_'
2018-03-05 20:17:47.013721 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting 'snapset'
2018-03-05 20:17:47.013726 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.013727 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head# = 0
2018-03-05 20:17:47.013732 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e14895f4:::.dir.default.6582062.5213.28:head seed ffffffff
2018-03-05 20:17:47.013749 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.013751 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head# 0~524288
2018-03-05 20:17:47.013783 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head# 0~0/524288
2018-03-05 20:17:47.013787 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head#
2018-03-05 20:17:47.013851 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e14895f4:::.dir.default.6582062.5213.28:head#
2018-03-05 20:17:47.013886 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e14895f4:::.dir.default.6582062.5213.28:head omap_digest f5109350
2018-03-05 20:17:47.013901 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.013920 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head# = 0 (size 0)
2018-03-05 20:17:47.013926 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head#
2018-03-05 20:17:47.032544 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting '_'
2018-03-05 20:17:47.032563 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting 'snapset'
2018-03-05 20:17:47.032569 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.032570 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head# = 0
2018-03-05 20:17:47.032575 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e14896aa:::.dir.default.6599779.35066.27:head seed ffffffff
2018-03-05 20:17:47.032593 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.032596 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head# 0~524288
2018-03-05 20:17:47.041300 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head# 0~0/524288
2018-03-05 20:17:47.041325 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head#
2018-03-05 20:17:47.041422 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e14896aa:::.dir.default.6599779.35066.27:head#
2018-03-05 20:17:47.041467 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e14896aa:::.dir.default.6599779.35066.27:head omap_digest f5109350
2018-03-05 20:17:47.041491 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.041512 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head# = 0 (size 0)
2018-03-05 20:17:47.041522 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head#
2018-03-05 20:17:47.056284 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 192 getting '_'
2018-03-05 20:17:47.056307 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 192 getting 'snapset'
2018-03-05 20:17:47.056313 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.056315 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head# = 0
2018-03-05 20:17:47.056321 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1489b28:::.dir.default.6586352.19527.6:head seed ffffffff
2018-03-05 20:17:47.056341 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.056343 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head# 0~524288
2018-03-05 20:17:47.056395 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head# 0~0/524288
2018-03-05 20:17:47.056401 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head#
2018-03-05 20:17:47.056478 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1489b28:::.dir.default.6586352.19527.6:head#
2018-03-05 20:17:47.056518 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1489b28:::.dir.default.6586352.19527.6:head omap_digest f5109350
2018-03-05 20:17:47.056534 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.056554 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head# = 0 (size 0)
2018-03-05 20:17:47.056561 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head#
2018-03-05 20:17:47.069362 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 36 getting '_'
2018-03-05 20:17:47.069383 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 36 getting 'snapset'
2018-03-05 20:17:47.069389 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.069390 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head# = 0
2018-03-05 20:17:47.069395 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1489b2b:::.dir.default.6614891.24114.16:head seed ffffffff
2018-03-05 20:17:47.069407 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.069409 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head# 0~524288
2018-03-05 20:17:47.069450 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head# 0~0/524288
2018-03-05 20:17:47.069462 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head#
2018-03-05 20:17:47.069519 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1489b2b:::.dir.default.6614891.24114.16:head#
2018-03-05 20:17:47.069554 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1489b2b:::.dir.default.6614891.24114.16:head omap_digest f5109350
2018-03-05 20:17:47.069570 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.069589 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head# = 0 (size 0)
2018-03-05 20:17:47.069595 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head#
2018-03-05 20:17:47.076446 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting '_'
2018-03-05 20:17:47.076466 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting 'snapset'
2018-03-05 20:17:47.076471 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.076472 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head# = 0
2018-03-05 20:17:47.076477 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1489bd4:::.dir.default.6614891.18413.9:head seed ffffffff
2018-03-05 20:17:47.076491 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.076493 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head# 0~524288
2018-03-05 20:17:47.076523 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head# 0~0/524288
2018-03-05 20:17:47.076528 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head#
2018-03-05 20:17:47.076594 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1489bd4:::.dir.default.6614891.18413.9:head#
2018-03-05 20:17:47.076630 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1489bd4:::.dir.default.6614891.18413.9:head omap_digest f5109350
2018-03-05 20:17:47.076645 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.076665 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head# = 0 (size 0)
2018-03-05 20:17:47.076671 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head#
2018-03-05 20:17:47.082277 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting '_'
2018-03-05 20:17:47.082296 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting 'snapset'
2018-03-05 20:17:47.082301 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.082303 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head# = 0
2018-03-05 20:17:47.082307 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e1489f73:::.dir.default.6708021.15007.27:head seed ffffffff
2018-03-05 20:17:47.082318 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.082320 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head# 0~524288
2018-03-05 20:17:47.082364 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head# 0~0/524288
2018-03-05 20:17:47.082369 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head#
2018-03-05 20:17:47.082430 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e1489f73:::.dir.default.6708021.15007.27:head#
2018-03-05 20:17:47.082465 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e1489f73:::.dir.default.6708021.15007.27:head omap_digest f5109350
2018-03-05 20:17:47.082480 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.082500 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head# = 0 (size 0)
2018-03-05 20:17:47.082506 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head#
2018-03-05 20:17:47.084841 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting '_'
2018-03-05 20:17:47.084861 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting 'snapset'
2018-03-05 20:17:47.084867 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.084869 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head# = 0
2018-03-05 20:17:47.084873 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148a44e:::.dir.default.6614891.3366.14:head seed ffffffff
2018-03-05 20:17:47.084887 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.084889 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head# 0~524288
2018-03-05 20:17:47.084929 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head# 0~0/524288
2018-03-05 20:17:47.084940 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head#
2018-03-05 20:17:47.084994 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148a44e:::.dir.default.6614891.3366.14:head#
2018-03-05 20:17:47.085028 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148a44e:::.dir.default.6614891.3366.14:head omap_digest f5109350
2018-03-05 20:17:47.085043 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.085063 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head# = 0 (size 0)
2018-03-05 20:17:47.085069 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head#
2018-03-05 20:17:47.103800 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting '_'
2018-03-05 20:17:47.103820 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting 'snapset'
2018-03-05 20:17:47.103825 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.103826 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head# = 0
2018-03-05 20:17:47.103831 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148a66f:::.dir.default.6586352.11407.1:head seed ffffffff
2018-03-05 20:17:47.103845 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.103846 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head# 0~524288
2018-03-05 20:17:47.103889 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head# 0~0/524288
2018-03-05 20:17:47.103894 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head#
2018-03-05 20:17:47.103947 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148a66f:::.dir.default.6586352.11407.1:head#
2018-03-05 20:17:47.103981 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148a66f:::.dir.default.6586352.11407.1:head omap_digest f5109350
2018-03-05 20:17:47.103996 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.104016 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head# = 0 (size 0)
2018-03-05 20:17:47.104023 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head#
2018-03-05 20:17:47.117912 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting '_'
2018-03-05 20:17:47.117928 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting 'snapset'
2018-03-05 20:17:47.117934 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.117935 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head# = 0
2018-03-05 20:17:47.117939 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148aa4b:::.dir.default.6614891.37088.27:head seed ffffffff
2018-03-05 20:17:47.117952 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.117953 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head# 0~524288
2018-03-05 20:17:47.117998 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head# 0~0/524288
2018-03-05 20:17:47.118011 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head#
2018-03-05 20:17:47.118066 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148aa4b:::.dir.default.6614891.37088.27:head#
2018-03-05 20:17:47.118101 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148aa4b:::.dir.default.6614891.37088.27:head omap_digest f5109350
2018-03-05 20:17:47.118116 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.118136 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head# = 0 (size 0)
2018-03-05 20:17:47.118142 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head#
2018-03-05 20:17:47.122199 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting '_'
2018-03-05 20:17:47.122218 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting 'snapset'
2018-03-05 20:17:47.122224 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.122226 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head# = 0
2018-03-05 20:17:47.122230 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148ab0b:::.dir.default.6599779.39400.1:head seed ffffffff
2018-03-05 20:17:47.122242 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.122244 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head# 0~524288
2018-03-05 20:17:47.122276 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head# 0~0/524288
2018-03-05 20:17:47.122281 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head#
2018-03-05 20:17:47.122342 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148ab0b:::.dir.default.6599779.39400.1:head#
2018-03-05 20:17:47.122376 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148ab0b:::.dir.default.6599779.39400.1:head omap_digest cc99ac09
2018-03-05 20:17:47.122391 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.122411 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head# = 0 (size 0)
2018-03-05 20:17:47.122418 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head#
2018-03-05 20:17:47.129494 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting '_'
2018-03-05 20:17:47.129513 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting 'snapset'
2018-03-05 20:17:47.129519 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.129520 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head# = 0
2018-03-05 20:17:47.129524 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148ae61:::.dir.default.6599779.14040.6:head seed ffffffff
2018-03-05 20:17:47.129536 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.129538 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head# 0~524288
2018-03-05 20:17:47.129598 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head# 0~0/524288
2018-03-05 20:17:47.129606 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head#
2018-03-05 20:17:47.129661 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148ae61:::.dir.default.6599779.14040.6:head#
2018-03-05 20:17:47.129695 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148ae61:::.dir.default.6599779.14040.6:head omap_digest f5109350
2018-03-05 20:17:47.129710 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.129729 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head# = 0 (size 0)
2018-03-05 20:17:47.129736 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head#
2018-03-05 20:17:47.137633 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 192 getting '_'
2018-03-05 20:17:47.137653 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 192 getting 'snapset'
2018-03-05 20:17:47.137658 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.137660 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head# = 0
2018-03-05 20:17:47.137664 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148af2e:::.dir.default.6586352.39617.10:head seed ffffffff
2018-03-05 20:17:47.137678 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.137680 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head# 0~524288
2018-03-05 20:17:47.137718 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head# 0~0/524288
2018-03-05 20:17:47.137723 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head#
2018-03-05 20:17:47.137784 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148af2e:::.dir.default.6586352.39617.10:head#
2018-03-05 20:17:47.137819 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148af2e:::.dir.default.6586352.39617.10:head omap_digest f5109350
2018-03-05 20:17:47.137834 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.137854 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head# = 0 (size 0)
2018-03-05 20:17:47.137860 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head#
2018-03-05 20:17:47.142505 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 36 getting '_'
2018-03-05 20:17:47.142524 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 36 getting 'snapset'
2018-03-05 20:17:47.142530 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.142532 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head# = 0
2018-03-05 20:17:47.142536 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148b020:::.dir.default.6599779.15359.29:head seed ffffffff
2018-03-05 20:17:47.142550 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.142552 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head# 0~524288
2018-03-05 20:17:47.142594 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head# 0~0/524288
2018-03-05 20:17:47.142602 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head#
2018-03-05 20:17:47.142657 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148b020:::.dir.default.6599779.15359.29:head#
2018-03-05 20:17:47.142690 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148b020:::.dir.default.6599779.15359.29:head omap_digest f5109350
2018-03-05 20:17:47.142705 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.142725 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head# = 0 (size 0)
2018-03-05 20:17:47.142731 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head#
2018-03-05 20:17:47.158845 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting '_'
2018-03-05 20:17:47.158864 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 199 getting 'snapset'
2018-03-05 20:17:47.158870 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.158871 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head# = 0
2018-03-05 20:17:47.158876 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148b1d4:::.dir.default.6582062.4652.23:head seed ffffffff
2018-03-05 20:17:47.158891 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.158893 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head# 0~524288
2018-03-05 20:17:47.158932 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head# 0~0/524288
2018-03-05 20:17:47.158944 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head#
2018-03-05 20:17:47.158997 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148b1d4:::.dir.default.6582062.4652.23:head#
2018-03-05 20:17:47.159032 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148b1d4:::.dir.default.6582062.4652.23:head omap_digest f5109350
2018-03-05 20:17:47.159047 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.159074 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head# = 0 (size 0)
2018-03-05 20:17:47.159080 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head#
2018-03-05 20:17:47.175853 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.175897 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.175907 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.175913 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process begin MSG
2018-03-05 20:17:47.175917 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process got MSG header
2018-03-05 20:17:47.175922 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process got envelope type=70 src osd.45 front=2004 data=0 off 0
2018-03-05 20:17:47.175928 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:47.175931 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.175932 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:47.175937 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:47.175942 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process begin MSG
2018-03-05 20:17:47.175945 7fb308ad1700 10 throttle(msgr_dispatch_throttler-hb_front_server 0x557b052b78f8) get_or_fail 2004 success (0 -> 2004)
2018-03-05 20:17:47.175947 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process got MSG header
2018-03-05 20:17:47.175948 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:47.175952 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_HEADER pgs=142 cs=1 l=1).process got envelope type=70 src osd.45 front=2004 data=0 off 0
2018-03-05 20:17:47.175955 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=142 cs=1 l=1).process got front 2004
2018-03-05 20:17:47.175958 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:47.175963 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:47.175966 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1).process aborted = 0
2018-03-05 20:17:47.175966 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:47.175972 7fb309ad3700 10 throttle(msgr_dispatch_throttler-hb_back_server 0x557b052b7ff8) get_or_fail 2004 success (0 -> 2004)
2018-03-05 20:17:47.175970 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1).process got 2004 + 0 + 0 byte message
2018-03-05 20:17:47.175975 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:47.175982 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=142 cs=1 l=1).process got front 2004
2018-03-05 20:17:47.175983 7fb308ad1700  5 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1). rx osd.45 seq 92 0x557b100a4200 osd_ping(ping e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.175987 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1).process aborted = 0
2018-03-05 20:17:47.175992 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1).process got 2004 + 0 + 0 byte message
2018-03-05 20:17:47.175992 7fb308ad1700  1 -- [XXXX::7108]:6813/48758 <== osd.45 [XXXX::7107]:0/53460 92 ==== osd_ping(ping e157808 stamp 2018-03-05 20:17:47.177604) v4 ==== 2004+0+0 (2922938168 0 0) 0x557b100a4200 con 0x557b0d663800
2018-03-05 20:17:47.176003 7fb308ad1700 20 heartbeat_map is_healthy = healthy, total workers: 11, number of unhealthy: 0
2018-03-05 20:17:47.176004 7fb309ad3700  5 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=142 cs=1 l=1). rx osd.45 seq 92 0x557b113c6e00 osd_ping(ping e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.176005 7fb308ad1700  1 -- [XXXX::7108]:6813/48758 --> [XXXX::7107]:0/53460 -- osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4 -- 0x557b100a4400 con 0
2018-03-05 20:17:47.176026 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).prepare_send_message m osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.176025 7fb309ad3700  1 -- [2607:f298:4:3245::7108]:6814/48758 <== osd.45 [2607:f298:4:3245::7107]:0/53460 92 ==== osd_ping(ping e157808 stamp 2018-03-05 20:17:47.177604) v4 ==== 2004+0+0 (2922938168 0 0) 0x557b113c6e00 con 0x557b0d665000
2018-03-05 20:17:47.176033 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).prepare_send_message encoding features 2305244844532236283 0x557b100a4400 osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.176045 7fb308ad1700 15 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).send_message inline write is denied, reschedule m=0x557b100a4400
2018-03-05 20:17:47.176053 7fb308ad1700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.45 [2607:f298:4:3245::7107]:6803/1053460 existing 0x557b0d5a5000
2018-03-05 20:17:47.176059 7fb308ad1700 20 osd.2 157808 share_map_peer 0x557b0d5a5000 already has epoch 157808
2018-03-05 20:17:47.176066 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 dispatch_throttle_release 2004 to dispatch throttler 2004/104857600
2018-03-05 20:17:47.176069 7fb308ad1700 10 throttle(msgr_dispatch_throttler-hb_front_server 0x557b052b78f8) put 2004 (2004 -> 0)
2018-03-05 20:17:47.176070 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 done calling dispatch on 0x557b100a4200
2018-03-05 20:17:47.176073 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:47.176072 7fb309ad3700 20 heartbeat_map is_healthy = healthy, total workers: 11, number of unhealthy: 0
2018-03-05 20:17:47.176079 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).handle_write
2018-03-05 20:17:47.176086 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).write_message signed m=0x557b100a4400): sig = 0
2018-03-05 20:17:47.176080 7fb309ad3700  1 -- [2607:f298:4:3245::7108]:6814/48758 --> [2607:f298:4:3245::7107]:0/53460 -- osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4 -- 0x557b10167c00 con 0
2018-03-05 20:17:47.176091 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending message type=70 src osd.2 front=2004 data=0 off 0
2018-03-05 20:17:47.176097 7fb308ad1700 20 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending 92 0x557b100a4400
2018-03-05 20:17:47.176095 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).prepare_send_message m osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.176103 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).prepare_send_message encoding features 2305244844532236283 0x557b10167c00 osd_ping(ping_reply e157808 stamp 2018-03-05 20:17:47.177604) v4
2018-03-05 20:17:47.176114 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1)._try_send sent bytes 2079 remaining bytes 0
2018-03-05 20:17:47.176120 7fb308ad1700 10 -- [XXXX::7108]:6813/48758 >> [XXXX::7107]:0/53460 conn(0x557b0d663800 :6813 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending 0x557b100a4400 done.
2018-03-05 20:17:47.176125 7fb309ad3700 15 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).send_message inline write is denied, reschedule m=0x557b10167c00
2018-03-05 20:17:47.176135 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6807/48758 get_connection osd.45 [2607:f298:4:3245::7107]:6803/1053460 existing 0x557b0d5a5000
2018-03-05 20:17:47.176141 7fb309ad3700 20 osd.2 157808 share_map_peer 0x557b0d5a5000 already has epoch 157808
2018-03-05 20:17:47.176145 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6814/48758 dispatch_throttle_release 2004 to dispatch throttler 2004/104857600
2018-03-05 20:17:47.176149 7fb309ad3700 10 throttle(msgr_dispatch_throttler-hb_back_server 0x557b052b7ff8) put 2004 (2004 -> 0)
2018-03-05 20:17:47.176151 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 done calling dispatch on 0x557b113c6e00
2018-03-05 20:17:47.176154 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:47.176165 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).handle_write
2018-03-05 20:17:47.176170 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).write_message signed m=0x557b10167c00): sig = 0
2018-03-05 20:17:47.176176 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending message type=70 src osd.2 front=2004 data=0 off 0
2018-03-05 20:17:47.176182 7fb309ad3700 20 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending 92 0x557b10167c00
2018-03-05 20:17:47.176206 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1)._try_send sent bytes 2079 remaining bytes 0
2018-03-05 20:17:47.176211 7fb309ad3700 10 -- [2607:f298:4:3245::7108]:6814/48758 >> [2607:f298:4:3245::7107]:0/53460 conn(0x557b0d665000 :6814 s=STATE_OPEN pgs=142 cs=1 l=1).write_message sending 0x557b10167c00 done.
2018-03-05 20:17:47.178508 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting '_'
2018-03-05 20:17:47.178529 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 195 getting 'snapset'
2018-03-05 20:17:47.178534 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.178535 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head# = 0
2018-03-05 20:17:47.178540 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148b217:::.dir.default.6708018.1204.8:head seed ffffffff
2018-03-05 20:17:47.178553 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.178555 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head# 0~524288
2018-03-05 20:17:47.178601 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head# 0~0/524288
2018-03-05 20:17:47.178608 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head#
2018-03-05 20:17:47.178669 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148b217:::.dir.default.6708018.1204.8:head#
2018-03-05 20:17:47.178704 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148b217:::.dir.default.6708018.1204.8:head omap_digest cc99ac09
2018-03-05 20:17:47.178719 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.178738 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head# = 0 (size 0)
2018-03-05 20:17:47.178752 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head#
2018-03-05 20:17:47.190942 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting '_'
2018-03-05 20:17:47.190961 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 43 getting 'snapset'
2018-03-05 20:17:47.190966 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.190968 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head# = 0
2018-03-05 20:17:47.190972 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148b6f5:::.dir.default.6586352.27495.8:head seed ffffffff
2018-03-05 20:17:47.190983 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.190986 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head# 0~524288
2018-03-05 20:17:47.191023 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head# 0~0/524288
2018-03-05 20:17:47.191028 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head#
2018-03-05 20:17:47.191098 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148b6f5:::.dir.default.6586352.27495.8:head#
2018-03-05 20:17:47.191133 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148b6f5:::.dir.default.6586352.27495.8:head omap_digest f5109350
2018-03-05 20:17:47.191149 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.191168 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head# = 0 (size 0)
2018-03-05 20:17:47.191174 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head#
2018-03-05 20:17:47.209463 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting '_'
2018-03-05 20:17:47.209484 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 197 getting 'snapset'
2018-03-05 20:17:47.209490 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.209491 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head# = 0
2018-03-05 20:17:47.209496 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148b930:::.dir.default.6708018.12222.28:head seed ffffffff
2018-03-05 20:17:47.209509 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.209511 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head# 0~524288
2018-03-05 20:17:47.209548 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head# 0~0/524288
2018-03-05 20:17:47.209554 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head#
2018-03-05 20:17:47.209626 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148b930:::.dir.default.6708018.12222.28:head#
2018-03-05 20:17:47.209663 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148b930:::.dir.default.6708018.12222.28:head omap_digest f5109350
2018-03-05 20:17:47.209678 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.209697 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head# = 0 (size 0)
2018-03-05 20:17:47.209704 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head#
2018-03-05 20:17:47.212674 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting '_'
2018-03-05 20:17:47.212691 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 189 getting 'snapset'
2018-03-05 20:17:47.212696 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.212698 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head# = 0
2018-03-05 20:17:47.212702 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148bb3c:::.dir.default.6614891.22856.5:head seed ffffffff
2018-03-05 20:17:47.212718 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.212720 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head# 0~524288
2018-03-05 20:17:47.212762 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head# 0~0/524288
2018-03-05 20:17:47.212767 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head#
2018-03-05 20:17:47.212824 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148bb3c:::.dir.default.6614891.22856.5:head#
2018-03-05 20:17:47.212855 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148bb3c:::.dir.default.6614891.22856.5:head omap_digest f5109350
2018-03-05 20:17:47.212871 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.212890 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head# = 0 (size 0)
2018-03-05 20:17:47.212896 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head#
2018-03-05 20:17:47.226507 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting '_'
2018-03-05 20:17:47.226526 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 185 getting 'snapset'
2018-03-05 20:17:47.226532 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.226534 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head# = 0
2018-03-05 20:17:47.226538 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148bd28:::.dir.default.6582062.4462.0:head seed ffffffff
2018-03-05 20:17:47.226552 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.226554 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head# 0~524288
2018-03-05 20:17:47.226597 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head# 0~0/524288
2018-03-05 20:17:47.226621 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head#
2018-03-05 20:17:47.226691 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148bd28:::.dir.default.6582062.4462.0:head#
2018-03-05 20:17:47.226741 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148bd28:::.dir.default.6582062.4462.0:head omap_digest f5109350
2018-03-05 20:17:47.226763 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.226785 7fb2f0710700 10 filestore(/srv/ceph/osd/2) stat(3200): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head# = 0 (size 0)
2018-03-05 20:17:47.226791 7fb2f0710700 15 filestore(/srv/ceph/osd/2) getattrs(4467): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head#
2018-03-05 20:17:47.245701 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting '_'
2018-03-05 20:17:47.245725 7fb2f0710700 20 filestore(/srv/ceph/osd/2) _fgetattrs(4334): 191 getting 'snapset'
2018-03-05 20:17:47.245731 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4489): no xattr exists in object_map r = 0
2018-03-05 20:17:47.245732 7fb2f0710700 10 filestore(/srv/ceph/osd/2) getattrs(4522): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head# = 0
2018-03-05 20:17:47.245737 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub 44:e148c6ac:::.dir.default.6599779.26330.4:head seed ffffffff
2018-03-05 20:17:47.245753 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.245755 7fb2f0710700 15 filestore(/srv/ceph/osd/2) read(3232): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head# 0~524288
2018-03-05 20:17:47.245822 7fb2f0710700 10 filestore(/srv/ceph/osd/2) read(3287): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head# 0~0/524288
2018-03-05 20:17:47.245830 7fb2f0710700 15 filestore(/srv/ceph/osd/2) omap_get_header(5073): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head#
2018-03-05 20:17:47.245900 7fb2f0710700 15 filestore(/srv/ceph/osd/2) get_omap_iterator(5189): 44.7_head/#44:e148c6ac:::.dir.default.6599779.26330.4:head#
2018-03-05 20:17:47.245939 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] be_deep_scrub  44:e148c6ac:::.dir.default.6599779.26330.4:head omap_digest f5109350
2018-03-05 20:17:47.246022 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[44.7( v 157803'3427203 (157803'3425666,157803'3427203] local-lis/les=157807/157808 n=189008 ec=154762/154762 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157036) [26,38,2] r=2 lpr=157807 luod=0'0 crt=157803'3427203 lcod 0'0 active] build_scrub_map_chunk done
2018-03-05 20:17:47.246075 7fb2f0710700  1 -- [2607:f298:4:3245::7108]:6807/48758 --> [2607:f298:4:3245::7097]:6804/64507 -- rep_scrubmap(44.7 e157808 from shard 2) v1 -- 0x557b113c5600 con 0
2018-03-05 20:17:47.246086 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).prepare_send_message m rep_scrubmap(44.7 e157808 from shard 2) v1
2018-03-05 20:17:47.246095 7fb2f0710700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).prepare_send_message encoding features 2305244844532236283 0x557b113c5600 rep_scrubmap(44.7 e157808 from shard 2) v1
2018-03-05 20:17:47.246139 7fb2f0710700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).send_message inline write is denied, reschedule m=0x557b113c5600
2018-03-05 20:17:47.246147 7fb2f0710700 20 Event(0x557b05026880 nevent=5000 time_id=37).wakeup
2018-03-05 20:17:47.246163 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b076e1000 finish
2018-03-05 20:17:47.246170 7fb2f0710700  6 -- op tracker --  seq: 1403, time: 2018-03-05 20:17:47.246169, event: done, op: replica scrub(pg: 44.7,from:0'0,to:0'0,epoch:157808/157807,start:44:e1488cb6:::.dir.default.6599779.41685.23:0,end:44:e148c9ca:::.dir.default.6614891.58717.17:0,chunky:1,deep:1,seed:4294967295,version:7)
2018-03-05 20:17:47.246163 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).handle_write
2018-03-05 20:17:47.246185 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.246192 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b0f475980 prio 196 cost 0 e157808) queued
2018-03-05 20:17:47.246199 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b0f475980 prio 196 cost 0 e157808) pg 0x557b05523000
2018-03-05 20:17:47.246203 7fb3092d2700 10 _calc_signature seq 1216 front_crc_ = 3864501369 middle_crc = 0 data_crc = 1547778157 sig = 9280144195182452221
2018-03-05 20:17:47.246206 7fb3092d2700 20 Putting signature in client message(seq # 1216): sig = 9280144195182452221
2018-03-05 20:17:47.246207 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).write_message signed m=0x557b113c5600): sig = 9280144195182452221
2018-03-05 20:17:47.246204 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b0f475980 prio 196 cost 0 latency 0.277367 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2 pg pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean]
2018-03-05 20:17:47.246214 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).write_message sending message type=117 src osd.2 front=39 data=10733 off 0
2018-03-05 20:17:47.246216 7fb2f0710700 20 osd.2 157808 share_map osd.14 [2607:f298:4:3245::7106]:6804/17339 157808
2018-03-05 20:17:47.246220 7fb2f0710700 20 osd.2 157808 should_share_map osd.14 [2607:f298:4:3245::7106]:6804/17339 157808
2018-03-05 20:17:47.246226 7fb2f0710700  6 -- op tracker --  seq: 1404, time: 2018-03-05 20:17:47.246225, event: reached_pg, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:47.246228 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).write_message sending 1216 0x557b113c5600
2018-03-05 20:17:47.246231 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] _handle_message: 0x557b0f475980
2018-03-05 20:17:47.246240 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_update_log_missing_reply got reply from 14
2018-03-05 20:17:47.246250 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b0f475980 finish
2018-03-05 20:17:47.246251 7fb2f0710700  6 -- op tracker --  seq: 1404, time: 2018-03-05 20:17:47.246251, event: done, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:47.246258 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.246260 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b105a3380 prio 196 cost 0 e157808) queued
2018-03-05 20:17:47.246263 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b105a3380 prio 196 cost 0 e157808) pg 0x557b05523000
2018-03-05 20:17:47.246267 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0)._try_send sent bytes 10847 remaining bytes 0
2018-03-05 20:17:47.246265 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b105a3380 prio 196 cost 0 latency 0.266268 pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1) v2 pg pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean]
2018-03-05 20:17:47.246273 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).write_message sending 0x557b113c5600 done.
2018-03-05 20:17:47.246274 7fb2f0710700 20 osd.2 157808 share_map osd.24 [2607:f298:4:3245::7097]:6800/25906 157808
2018-03-05 20:17:47.246277 7fb2f0710700 20 osd.2 157808 should_share_map osd.24 [2607:f298:4:3245::7097]:6800/25906 157808
2018-03-05 20:17:47.246281 7fb2f0710700  6 -- op tracker --  seq: 1405, time: 2018-03-05 20:17:47.246281, event: reached_pg, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:47.246284 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] _handle_message: 0x557b105a3380
2018-03-05 20:17:47.246292 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] do_update_log_missing_reply got reply from 24
2018-03-05 20:17:47.246300 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 luod=157808'772973 crt=157803'772972 lcod 157803'772972 mlcod 0'0 active+clean] repop_all_committed: repop tid 1 all committed 
2018-03-05 20:17:47.246307 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean] eval_repop repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0)
2018-03-05 20:17:47.246316 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]  commit: repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0)
2018-03-05 20:17:47.246325 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]  applied: repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0) 
2018-03-05 20:17:47.246338 7fb2f0710700 15 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean] publish_stats_to_osd 157808:780119
2018-03-05 20:17:47.246346 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]  removing repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0)
2018-03-05 20:17:47.246354 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]    q front is repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0)
2018-03-05 20:17:47.246363 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean] finished operator() r=0
2018-03-05 20:17:47.246370 7fb2f0710700 10 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]  sending commit on osd_op(client.9244044.0:72 38.c 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] snapc 0=[] ondisk+write+known_if_redirected e157808) v8 0x557b0d67b8c0
2018-03-05 20:17:47.246384 7fb2f0710700  1 -- [XXXX::7108]:6806/48758 --> [XXXX::7930]:0/51387864 -- osd_op_reply(72 usage.3 [call rgw.user_usage_log_trim] v157808'772974 uv263780 ondisk = 0) v8 -- 0x557b0d67b8c0 con 0
2018-03-05 20:17:47.246395 7fb2f0710700 15 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).send_message inline write is denied, reschedule m=0x557b0d67b8c0
2018-03-05 20:17:47.246400 7fb2f0710700 20 Event(0x557b05026880 nevent=5000 time_id=37).wakeup
2018-03-05 20:17:47.246405 7fb2f0710700 20 osd.2 pg_epoch: 157808 pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean] remove_repop repgather(0x557b0f3dcac0 157808'772974 rep_tid=1 committed?=1 applied?=1 r=0)
2018-03-05 20:17:47.246410 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).handle_write
2018-03-05 20:17:47.246419 7fb2f0710700  6 -- op tracker --  seq: 1402, time: 2018-03-05 20:17:47.246418, event: done, op: osd_op(client.9244044.0:72 38.c 38:3351fad7:::usage.3:head [call rgw.user_usage_log_trim] snapc 0=[] ondisk+write+known_if_redirected e157808)
2018-03-05 20:17:47.246422 7fb2f0710700 10 throttle(osd_client_bytes 0x557b05041c00) put 48 (48 -> 0)
2018-03-05 20:17:47.246418 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).prepare_send_message m osd_op_reply(72 usage.3 [call rgw.user_usage_log_trim] v157808'772974 uv263780 ondisk = 0) v8
2018-03-05 20:17:47.246427 7fb2f0710700 10 throttle(osd_client_bytes 0x557b05041c00) put 0 (0 -> 0)
2018-03-05 20:17:47.246433 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b105a3380 finish
2018-03-05 20:17:47.246428 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).prepare_send_message encoding features 2305244844532236283 0x557b0d67b8c0 osd_op_reply(72 usage.3 [call rgw.user_usage_log_trim] v157808'772974 uv263780 ondisk = 0) v8
2018-03-05 20:17:47.246435 7fb2f0710700  6 -- op tracker --  seq: 1405, time: 2018-03-05 20:17:47.246435, event: done, op: pg_update_log_missing_reply(38.c epoch 157808/157807 rep_tid 1)
2018-03-05 20:17:47.246440 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 150
2018-03-05 20:17:47.246442 7fb2f0710700 20 osd.2 op_wq(2) _process empty q, waiting
2018-03-05 20:17:47.246444 7fb2f0710700 20 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fb2f0710700' grace 60 suicide 0
2018-03-05 20:17:47.246448 7fb3092d2700 10 _calc_signature seq 2 front_crc_ = 2395895143 middle_crc = 0 data_crc = 0 sig = 11415306493330140018
2018-03-05 20:17:47.246450 7fb3092d2700 20 Putting signature in client message(seq # 2): sig = 11415306493330140018
2018-03-05 20:17:47.246451 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).write_message signed m=0x557b0d67b8c0): sig = 11415306493330140018
2018-03-05 20:17:47.246458 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).write_message sending message type=43 src osd.2 front=151 data=0 off 0
2018-03-05 20:17:47.246463 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).write_message sending 2 0x557b0d67b8c0
2018-03-05 20:17:47.246476 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1)._try_send sent bytes 226 remaining bytes 0
2018-03-05 20:17:47.246485 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).write_message sending 0x557b0d67b8c0 done.
2018-03-05 20:17:47.246559 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:47.246570 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_TAG_ACK pgs=2192 cs=1 l=0).process prev state is STATE_OPEN
2018-03-05 20:17:47.246575 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_TAG_ACK pgs=2192 cs=1 l=0).process got ACK
2018-03-05 20:17:47.246582 7fb3092d2700 15 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_TAG_ACK pgs=2192 cs=1 l=0).handle_ack got ack seq 1216
2018-03-05 20:17:47.246587 7fb3092d2700 10 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN_TAG_ACK pgs=2192 cs=1 l=0).handle_ack got ack seq 1216 >= 1216 on 0x557b113c5600 rep_scrubmap(44.7 e157808 from shard 2) v1
2018-03-05 20:17:47.246616 7fb3092d2700 20 -- [2607:f298:4:3245::7108]:6807/48758 >> [2607:f298:4:3245::7097]:6804/64507 conn(0x557b0d537000 :-1 s=STATE_OPEN pgs=2192 cs=1 l=0).process prev state is STATE_OPEN_TAG_ACK
2018-03-05 20:17:47.247237 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.247261 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_HEADER pgs=19 cs=1 l=1).process prev state is STATE_OPEN
2018-03-05 20:17:47.247266 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_HEADER pgs=19 cs=1 l=1).process begin MSG
2018-03-05 20:17:47.247271 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_HEADER pgs=19 cs=1 l=1).process got MSG header
2018-03-05 20:17:47.247275 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_HEADER pgs=19 cs=1 l=1).process got envelope type=42 src client.9244044 front=216 data=48 off 0
2018-03-05 20:17:47.247281 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_THROTTLE_MESSAGE pgs=19 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_HEADER
2018-03-05 20:17:47.247285 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=19 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_MESSAGE
2018-03-05 20:17:47.247290 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_THROTTLE_BYTES pgs=19 cs=1 l=1).process wants 264 bytes from policy throttler 0/524288000
2018-03-05 20:17:47.247299 7fb3092d2700 10 throttle(osd_client_bytes 0x557b05041c00) get_or_fail 264 success (0 -> 264)
2018-03-05 20:17:47.247301 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE pgs=19 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_BYTES
2018-03-05 20:17:47.247306 7fb3092d2700 10 throttle(msgr_dispatch_throttler-client 0x557b052b63f8) get_or_fail 264 success (0 -> 264)
2018-03-05 20:17:47.247307 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=19 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_THROTTLE_DISPATCH_QUEUE
2018-03-05 20:17:47.247313 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_FRONT pgs=19 cs=1 l=1).process got front 216
2018-03-05 20:17:47.247317 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_DATA_PREPARE pgs=19 cs=1 l=1).process allocating new rx buffer at offset 0
2018-03-05 20:17:47.247322 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1).process aborted = 0
2018-03-05 20:17:47.247326 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1).process got 216 + 0 + 48 byte message
2018-03-05 20:17:47.247352 7fb3092d2700 10 _calc_signature seq 3 front_crc_ = 2889505296 middle_crc = 0 data_crc = 316140335 sig = 4551475545586765660
2018-03-05 20:17:47.247355 7fb3092d2700  5 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=19 cs=1 l=1). rx client.9244044 seq 3 0x557b0d67b8c0 osd_op(client.9244044.0:73 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808) v8
2018-03-05 20:17:47.247367 7fb3092d2700  1 -- [XXXX::7108]:6806/48758 <== client.9244044 [XXXX::7930]:0/51387864 3 ==== osd_op(client.9244044.0:73 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808) v8 ==== 216+0+48 (2889505296 0 316140335) 0x557b0d67b8c0 con 0x557b0eb83000
2018-03-05 20:17:47.247377 7fb3092d2700 15 osd.2 157808 enqueue_op 0x557b076e1780 prio 63 cost 48 latency 0.000110 epoch 157808 osd_op(client.9244044.0:73 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808) v8
2018-03-05 20:17:47.247382 7fb3092d2700  6 -- op tracker --  seq: 1406, time: 2018-03-05 20:17:47.247382, event: queued_for_pg, op: osd_op(client.9244044.0:73 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808)
2018-03-05 20:17:47.247392 7fb3092d2700 20 osd.2 op_wq(2) _enqueue 38.c PGQueueable(0x557b076e1780 prio 63 cost 48 e157808)
2018-03-05 20:17:47.247400 7fb3092d2700 10 -- [XXXX::7108]:6806/48758 dispatch_throttle_release 264 to dispatch throttler 264/104857600
2018-03-05 20:17:47.247404 7fb3092d2700 10 throttle(msgr_dispatch_throttler-client 0x557b052b63f8) put 264 (264 -> 0)
2018-03-05 20:17:47.247406 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 done calling dispatch on 0x557b0d67b8c0
2018-03-05 20:17:47.247409 7fb3092d2700 20 -- [XXXX::7108]:6806/48758 >> [XXXX::7930]:0/51387864 conn(0x557b0eb83000 :6806 s=STATE_OPEN pgs=19 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2018-03-05 20:17:47.247414 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b076e1780 prio 63 cost 48 e157808) queued
2018-03-05 20:17:47.247423 7fb2f0710700 20 osd.2 op_wq(2) _process 38.c item PGQueueable(0x557b076e1780 prio 63 cost 48 e157808) pg 0x557b05523000
2018-03-05 20:17:47.247425 7fb2f0710700 10 osd.2 157808 dequeue_op 0x557b076e1780 prio 63 cost 48 latency 0.000159 osd_op(client.9244044.0:73 38.c 38.eb5f8acc (undecoded) ondisk+write+known_if_redirected e157808) v8 pg pg[38.c( v 157808'772974 (157803'740282,157808'772974] local-lis/les=157807/157808 n=1 ec=154756/154756 lis/c 157807/157807 les/c/f 157808/157808/0 157807/157807/157807) [2,14,24] r=0 lpr=157807 crt=157803'772972 lcod 157808'772973 mlcod 0'0 active+clean]