Ceph » RADOS

ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_replicas(10)
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_primary recovering 0 in pg, missing missing(0 may_include_deletes = 1)
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_primary {}
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_backfill (10) bft=7(2) last_backfill_started 2:f7b829c8:::smithi16613012-16:2db
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] peer osd.7(2) info 2.fs2( v 874'1021 lc 856'1013 (464'631,874'1021] lb 2:f7b829c8:::smithi16613012-16:2db local-lis/les=833/834 n=6 ec=20/20 lis/c 833/680 les/c/f 834/681/0 833/833/619) 
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] update_range: bi is current 
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]]    my backfill interval BackfillInfo(2:f7b829c8:::smithi16613012-16:head-MAX 1 objects {2:f7b829c8:::smithi16613012-16:head=824'1009})
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]]  peer shard 7(2) backfill BackfillInfo(MAX-MAX 0 objects)
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] get_object_context: found obc in cache: 0x55b491e44940
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] get_object_context: 0x55b491e44940 2:f7b829c8:::smithi16613012-16:head rwstate(write n=1 w=0) oi: 2:f7b829c8:::smithi16613012-16:head(824'1009 client.4308.0:8771 dirty|data_digest s 1386
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] backfill blocking on 2:f7b829c8:::smithi16613012-16:head; could not get rw_manager lock
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] backfill_pos is 2:f7b829c8:::smithi16613012-16:head
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] starting new_last_backfill at 2:f7b829c8:::smithi16613012-16:2db
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] possible new_last_backfill at 2:f7b829c8:::smithi16613012-16:2db
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] final new_last_backfill at 2:f7b829c8:::smithi16613012-16:2db
ing mbc={0={},1={},2={}} trimq=[2d7~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]]  started 0
...
~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_replicas(10)
~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_primary recovering 0 in pg, missing missing(0 may_include_deletes = 1)
~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_primary {}
~1,2da~1,2dc~1,2e1~2,2e4~3,2e8~1,2ea~1,2ed~2,2f1~2,2f4~2,2f7~1,2f9~2,2fd~2,301~1,307~2,30b~2]] recover_backfill (10) bft=7(2) last_backfill_started 2:f7b829c8:::smithi16613012-16:2db

and it loops.
/a/sage-2019-07-10_01:52:27-rados-wip-sage-testing-2019-07-09-1801-distro-basic-smithi/4107563 osd.4 pg 2.f