2019-04-20 14:11:54.550 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0(unlocked)] enter Initial 2019-04-20 14:11:54.550 7fdb96931700 20 osd.0 pg_epoch: 9 pg[1.0(unlocked)] enter NotTrimming 2019-04-20 14:11:54.550 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c 0/0 les/c/f 0/0/0 0/0/0) [] r=-1 lpr=0 crt=0'0 unknown mbc={}] init role 1 up [1,0] acting [1,0] history ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9 past_intervals ([0,0] intervals=) 2019-04-20 14:11:54.550 7fdb96931700 20 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=0 crt=0'0 unknown mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 0) 2019-04-20 14:11:54.550 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=0 crt=0'0 unknown mbc={}] handle_initialize 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=0 crt=0'0 unknown mbc={}] exit Initial 0.000505 0 0.000000 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=0 crt=0'0 unknown mbc={}] enter Reset 2019-04-20 14:11:54.554 7fdb96931700 20 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=0 crt=0'0 unknown mbc={}] set_last_peering_reset 9 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] handle_activate_map 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] take_waiters 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] exit Reset 0.000133 1 0.000156 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] enter Started 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] enter Start 2019-04-20 14:11:54.554 7fdb96931700 1 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] state: transitioning to Stray 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] exit Start 0.000048 0 0.000000 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] enter Started/Stray 2019-04-20 14:11:54.554 7fdb96931700 20 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] handle_activate_map: Not dirtying info: last_persisted is 9 while current is 9 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 10 handle_pg_create_info new pg pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 9 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] handle_advance_map [1,0]/[1,0] -- 1/1 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] state: Started advmap 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] on_pool_change 2019-04-20 14:11:54.554 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] hit_set_clear 2019-04-20 14:11:54.554 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] agent_stop 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] handle_activate_map 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] take_waiters 2019-04-20 14:11:54.554 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] handle_activate_map: Not dirtying info: last_persisted is 9 while current is 10 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] do_peering_event: epoch_sent: 10 epoch_requested: 9 MLogRec from 1 +create_info 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] state: got info+log from osd.1 1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) log((0'0,0'0], crt=0'0) 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] exit Started/Stray 0.001772 3 0.000262 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] enter Started/ReplicaActive 2019-04-20 14:11:54.554 7fdb96931700 5 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] enter Started/ReplicaActive/RepNotRecovering 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=0/0 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] state: In ReplicaActive, about to call activate 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] activate - no missing, moving last_complete 0'0 -> 0'0 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] state: Activate Finished 2019-04-20 14:11:54.554 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] do_peering_event: epoch_sent: 10 epoch_requested: 10 NullEvt 2019-04-20 14:11:54.562 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 unknown mbc={}] _activate_committed 10 telling primary 2019-04-20 14:11:54.566 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 0/0 les/c/f 0/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] do_peering_event: epoch_sent: 10 epoch_requested: 10 MInfoRec from 1 info: 1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/0 les/c/f 10/0/0 9/9/9) 2019-04-20 14:11:54.566 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/0 les/c/f 10/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] update_history advanced history from ec=9/9 lis/c 9/0 les/c/f 10/0/0 9/9/9 2019-04-20 14:11:54.566 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/0 les/c/f 10/0/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] do_peering_event: epoch_sent: 10 epoch_requested: 10 MInfoRec from 1 info: 1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) 2019-04-20 14:11:54.566 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] update_history advanced history from ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9 2019-04-20 14:11:54.566 7fdb96931700 20 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] update_history clearing past_intervals 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 10 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_advance_map [1,0]/[1,0] -- 1/1 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] state: Started advmap 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] on_pool_change 2019-04-20 14:11:55.574 7fdb96931700 20 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] hit_set_clear 2019-04-20 14:11:55.574 7fdb96931700 20 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] agent_stop 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_activate_map 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] take_waiters 2019-04-20 14:11:55.574 7fdb96931700 20 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_activate_map: Not dirtying info: last_persisted is 10 while current is 11 2019-04-20 14:11:55.574 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] do_peering_event: epoch_sent: 11 epoch_requested: 11 NullEvt 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 11 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_advance_map [1,0]/[1,0] -- 1/1 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] state: Started advmap 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_activate_map 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] take_waiters 2019-04-20 14:12:03.826 7fdb96931700 20 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] handle_activate_map: Not dirtying info: last_persisted is 10 while current is 12 2019-04-20 14:12:03.826 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] do_peering_event: epoch_sent: 12 epoch_requested: 12 NullEvt 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 12 dequeue_op 0x561d71db83c0 prio 127 cost 957 latency 0.000266 osd_repop(client.4139.0:1 1.0 e12/9) v2 pg pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] _handle_message: 0x561d71db83c0 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=0 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] do_repop 1:ad551702:::test:head v 12'1 (transaction) 144 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( empty local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 crt=0'0 active mbc={}] append_log log((0'0,0'0], crt=0'0) [12'1 (0'0) modify 1:ad551702:::test:head by client.4139.0:1 2019-04-20 14:12:11.032054 0] 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=0'0 lcod 0'0 active mbc={}] add_log_entry 12'1 (0'0) modify 1:ad551702:::test:head by client.4139.0:1 2019-04-20 14:12:11.032054 0 2019-04-20 14:12:11.038 7fdb96931700 20 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] rollforward: entry=12'1 (0'0) modify 1:ad551702:::test:head by client.4139.0:1 2019-04-20 14:12:11.032054 0 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] append_log approx pg log length = 1 2019-04-20 14:12:11.038 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] append_log transaction_applied = 1 2019-04-20 14:12:11.046 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] repop_commit on op osd_repop(client.4139.0:1 1.0 e12/9 1:ad551702:::test:head v 12'1) v2, sending commit to osd.1 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 12 dequeue_op 0x561d71db9400 prio 127 cost 885 latency 0.000281 osd_repop(client.4142.0:1 1.0 e12/9) v2 pg pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] _handle_message: 0x561d71db9400 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] do_repop 1:ad551702:::test:head v 12'2 (transaction) 144 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] append_log log((0'0,12'1], crt=12'1) [12'2 (12'1) modify 1:ad551702:::test:head by client.4142.0:1 2019-04-20 14:12:21.503212 0] 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'1 lcod 0'0 active mbc={}] add_log_entry 12'2 (12'1) modify 1:ad551702:::test:head by client.4142.0:1 2019-04-20 14:12:21.503212 0 2019-04-20 14:12:21.506 7fdb96931700 20 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'2 lcod 0'0 active mbc={}] rollforward: entry=12'2 (12'1) modify 1:ad551702:::test:head by client.4142.0:1 2019-04-20 14:12:21.503212 0 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'2 lcod 0'0 active mbc={}] append_log approx pg log length = 2 2019-04-20 14:12:21.506 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'2 lcod 0'0 active mbc={}] append_log transaction_applied = 1 2019-04-20 14:12:21.514 7fdb96931700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'2 lcod 0'0 active mbc={}] repop_commit on op osd_repop(client.4142.0:1 1.0 e12/9 1:ad551702:::test:head v 12'2) v2, sending commit to osd.1 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] on_shutdown 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] cancel_copy_ops 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] cancel_flush_ops 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] cancel_proxy_ops 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] clear_backoffs 2019-04-20 14:12:29.549 7fdbb1411700 20 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] exit NotTrimming 2019-04-20 14:12:29.549 7fdbb1411700 20 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] enter NotTrimming 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] on_change 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] clear_async_reads 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] clear_primary_state 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:12:29.549 7fdbb1411700 20 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] agent_stop 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] cancel_recovery 2019-04-20 14:12:29.549 7fdbb1411700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=9 DELETING luod=0'0 lua=0'0 crt=12'2 lcod 12'1 active mbc={}] clear_recovery_state 2019-04-20 14:14:19.795 7f9e35104ec0 5 osd.0 pg_epoch: 12 pg[1.0(unlocked)] enter Initial 2019-04-20 14:14:19.795 7f9e35104ec0 20 osd.0 pg_epoch: 12 pg[1.0(unlocked)] enter NotTrimming 2019-04-20 14:14:19.799 7f9e35104ec0 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=0 crt=12'2 lcod 0'0 unknown mbc={}] handle_initialize 2019-04-20 14:14:19.799 7f9e35104ec0 5 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=0 crt=12'2 lcod 0'0 unknown mbc={}] exit Initial 0.001412 0 0.000000 2019-04-20 14:14:19.799 7f9e35104ec0 5 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=0 crt=12'2 lcod 0'0 unknown mbc={}] enter Reset 2019-04-20 14:14:19.799 7f9e35104ec0 20 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=0 crt=12'2 lcod 0'0 unknown mbc={}] set_last_peering_reset 12 2019-04-20 14:14:19.799 7f9e35104ec0 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:14:19.799 7f9e35104ec0 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:14:19.803 7f9e35104ec0 10 osd.0 12 load_pgs loaded pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] 2019-04-20 14:14:19.819 7f9e11fdf700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] do_peering_event: epoch_sent: 12 epoch_requested: 12 NullEvt 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 12 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] handle_advance_map []/[] -- -1/-1 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] state: Reset advmap 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] new interval newup [] newacting [] 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] state: should restart peering, calling start_peering_interval again 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=12 crt=12'2 lcod 0'0 unknown mbc={}] set_last_peering_reset 13 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=13 crt=12'2 lcod 0'0 unknown mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [1,0] r=1 lpr=13 crt=12'2 lcod 0'0 unknown mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] start_peering_interval: check_new_interval output: check_new_interval interval(9-12 up [1,0](1) acting [1,0](1)) up_thru 9 up_from 8 last_epoch_clean 10 interval(9-12 up [1,0](1) acting [1,0](1) maybe_went_rw) : primary up 8-9 includes interval 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 9/9/9) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] noting past ([9,12] intervals=([9,12] acting 0,1)) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2019-04-20 14:14:20.639 7f9e11fdf700 1 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] start_peering_interval up [1,0] -> [], acting [1,0] -> [], acting_primary 1 -> -1, up_primary 1 -> -1, role 1 -> -1, features acting 4540138292836696063 upacting 4540138292836696063 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] clear_primary_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] agent_stop 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] cancel_copy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] cancel_flush_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] cancel_proxy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] on_change_cleanup 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] exit NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown mbc={}] enter NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_role_change 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_recovery 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] clear_recovery_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 13 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_advance_map [1]/[1] -- 1/1 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: Reset advmap 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] new interval newup [1] newacting [1] 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: should restart peering, calling start_peering_interval again 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=13 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] set_last_peering_reset 14 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=14 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [] r=-1 lpr=14 pi=[9,13)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval: check_new_interval output: check_new_interval interval(13-13 up [](-1) acting [](-1)) : acting set is too small 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 13/13/13) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] noting past ([9,13] intervals=([9,12] acting 0,1)) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2019-04-20 14:14:20.639 7f9e11fdf700 1 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [] -> [1], acting [] -> [1], acting_primary ? -> 1, up_primary ? -> 1, role -1 -> -1, features acting 4540138292836696063 upacting 4540138292836696063 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] clear_primary_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] agent_stop 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_copy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_flush_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_proxy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change_cleanup 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] exit NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] enter NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] [] -> [1], acting primary ? -> 1 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_recovery 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] clear_recovery_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 14 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_advance_map [1]/[1] -- 1/1 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 15 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: Reset advmap 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 15 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 15 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_advance_map [1,0]/[1,0] -- 1/1 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: Reset advmap 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] new interval newup [1,0] newacting [1,0] 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: should restart peering, calling start_peering_interval again 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=14 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] set_last_peering_reset 16 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=16 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1] r=-1 lpr=16 pi=[9,14)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval: check_new_interval output: check_new_interval interval(14-15 up [1](1) acting [1](1)) up_thru 14 up_from 14 last_epoch_clean 10 interval(14-15 up [1](1) acting [1](1) maybe_went_rw) : primary up 14-14 includes interval 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 14/14/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] noting past ([9,15] intervals=([14,15] acting 1)) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(0 may_include_deletes = 1) 2019-04-20 14:14:20.639 7f9e11fdf700 1 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [1] -> [1,0], acting [1] -> [1,0], acting_primary 1 -> 1, up_primary 1 -> 1, role -1 -> 1, features acting 4540138292836696063 upacting 4540138292836696063 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] clear_primary_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] agent_stop 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_copy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_flush_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_proxy_ops 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change_cleanup 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_change 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] exit NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] enter NotTrimming 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] on_role_change 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] cancel_recovery 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] clear_recovery_state 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_activate_map 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] take_waiters 2019-04-20 14:14:20.639 7f9e11fdf700 5 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] exit Reset 0.842422 6 0.002299 2019-04-20 14:14:20.639 7f9e11fdf700 5 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] enter Started 2019-04-20 14:14:20.639 7f9e11fdf700 5 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] enter Start 2019-04-20 14:14:20.639 7f9e11fdf700 1 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: transitioning to Stray 2019-04-20 14:14:20.639 7f9e11fdf700 5 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] exit Start 0.000035 0 0.000000 2019-04-20 14:14:20.639 7f9e11fdf700 5 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] enter Started/Stray 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_activate_map: Not dirtying info: last_persisted is 12 while current is 16 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 9/9 les/c/f 10/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] do_peering_event: epoch_sent: 16 epoch_requested: 16 MQuery 1.0 from 1 query_epoch 16 query: query(info 0'0 epoch_sent 16) 2019-04-20 14:14:20.639 7f9e11fdf700 20 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] update_history advanced history from ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14 2019-04-20 14:14:20.639 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] sending info 2019-04-20 14:14:20.643 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] do_peering_event: epoch_sent: 16 epoch_requested: 16 NullEvt 2019-04-20 14:14:20.643 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] do_peering_event: epoch_sent: 16 epoch_requested: 16 MQuery 1.0 from 1 query_epoch 16 query: query(log 10'0 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14 epoch_sent 16) 2019-04-20 14:14:20.643 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] log request from 1 2019-04-20 14:14:20.643 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] sending info+missing+log since 10'0 2019-04-20 14:14:20.643 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] sending log((0'0,12'2], crt=12'2) missing(0 may_include_deletes = 1) 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 16 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_advance_map [1,0]/[1,0] -- 1/1 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: Started advmap 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_activate_map 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] take_waiters 2019-04-20 14:14:21.659 7f9e11fdf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] handle_activate_map: Not dirtying info: last_persisted is 16 while current is 17 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 MInfoRec from 1 info: 1.0( v 12'1 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown NOTIFY mbc={}] state: got info from osd.1 1.0( v 12'1 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) 2019-04-20 14:14:21.659 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 lcod 0'0 unknown NOTIFY m=1 mbc={}] exit Started/Stray 1.017375 6 0.000502 2019-04-20 14:14:21.659 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 lcod 0'0 unknown NOTIFY m=1 mbc={}] enter Started/ReplicaActive 2019-04-20 14:14:21.659 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 lcod 0'0 unknown NOTIFY m=1 mbc={}] enter Started/ReplicaActive/RepNotRecovering 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=9/10 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 lcod 0'0 unknown NOTIFY m=1 mbc={}] state: In ReplicaActive, about to call activate 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 lcod 0'0 unknown m=1 mbc={}] activate - not complete, missing(1 may_include_deletes = 1) 2019-04-20 14:14:21.659 7f9e11fdf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 unknown m=1 mbc={}] rollforward: entry=12'1 (0'0) modify 1:ad551702:::test:head by client.4139.0:1 2019-04-20 14:12:11.032054 0 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 unknown m=1 mbc={}] state: Activate Finished 2019-04-20 14:14:21.659 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 unknown m=1 mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 NullEvt 2019-04-20 14:14:21.663 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 crt=12'1 unknown m=1 mbc={}] _activate_committed 17 telling primary 2019-04-20 14:14:21.667 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 14/9 les/c/f 15/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 MInfoRec from 1 info: 1.0( v 12'1 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) 2019-04-20 14:14:21.667 7f9e11fdf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] update_history advanced history from ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14 2019-04-20 14:14:21.667 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 RequestRecoveryPrio: priority 180 2019-04-20 14:14:21.667 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.009077 3 0.000399 2019-04-20 14:14:21.667 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved 2019-04-20 14:14:21.667 7f9e11fdf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 RemoteRecoveryReserved 2019-04-20 14:14:21.667 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] exit Started/ReplicaActive/RepWaitRecoveryReserved 0.000422 1 0.000179 2019-04-20 14:14:21.667 7f9e11fdf700 5 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] enter Started/ReplicaActive/RepRecovering 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] on_shutdown 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] cancel_copy_ops 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] cancel_flush_ops 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] cancel_proxy_ops 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] clear_backoffs 2019-04-20 14:16:11.716 7f9e2cabf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] exit NotTrimming 2019-04-20 14:16:11.716 7f9e2cabf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] enter NotTrimming 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] on_change 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] clear_async_reads 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] clear_primary_state 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:16:11.716 7f9e2cabf700 20 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] agent_stop 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] cancel_recovery 2019-04-20 14:16:11.716 7f9e2cabf700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'1 lc 0'0 (0'0,12'1] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=16 DELETING pi=[9,16)/1 luod=0'0 crt=12'1 active m=1 mbc={}] clear_recovery_state 2019-04-20 14:16:34.400 7f46793a5ec0 5 osd.0 pg_epoch: 17 pg[1.0(unlocked)] enter Initial 2019-04-20 14:16:34.400 7f46793a5ec0 20 osd.0 pg_epoch: 17 pg[1.0(unlocked)] enter NotTrimming 2019-04-20 14:16:34.404 7f46793a5ec0 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=0 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] handle_initialize 2019-04-20 14:16:34.404 7f46793a5ec0 5 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=0 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] exit Initial 0.000914 0 0.000000 2019-04-20 14:16:34.404 7f46793a5ec0 5 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=0 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] enter Reset 2019-04-20 14:16:34.404 7f46793a5ec0 20 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=0 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] set_last_peering_reset 17 2019-04-20 14:16:34.404 7f46793a5ec0 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:16:34.404 7f46793a5ec0 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:16:34.404 7f46793a5ec0 10 osd.0 17 load_pgs loaded pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] 2019-04-20 14:16:34.432 7f4656280700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] do_peering_event: epoch_sent: 17 epoch_requested: 17 NullEvt 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 17 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] handle_advance_map []/[] -- -1/-1 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] state: Reset advmap 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] new interval newup [] newacting [] 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] state: should restart peering, calling start_peering_interval again 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=17 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] set_last_peering_reset 18 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=18 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [1,0] r=1 lpr=18 pi=[9,16)/1 crt=12'2 lcod 0'0 unknown m=1 mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] start_peering_interval: check_new_interval output: check_new_interval interval(16-17 up [1,0](1) acting [1,0](1)) up_thru 16 up_from 14 last_epoch_clean 10 interval(16-17 up [1,0](1) acting [1,0](1) maybe_went_rw) : primary up 14-16 includes interval 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 16/16/14) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] noting past ([9,17] intervals=([14,15] acting 1),([16,17] acting 0,1)) 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(1 may_include_deletes = 1) 2019-04-20 14:16:34.520 7f4656280700 1 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] start_peering_interval up [1,0] -> [], acting [1,0] -> [], acting_primary 1 -> -1, up_primary 1 -> -1, role 1 -> -1, features acting 4540138292836696063 upacting 4540138292836696063 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] clear_primary_state 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] agent_stop 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] on_change 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] cancel_copy_ops 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] cancel_flush_ops 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] cancel_proxy_ops 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] on_change_cleanup 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] on_change 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] exit NotTrimming 2019-04-20 14:16:34.520 7f4656280700 20 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown m=1 mbc={}] enter NotTrimming 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] on_role_change 2019-04-20 14:16:34.520 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] cancel_recovery 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] clear_recovery_state 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 18 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] handle_advance_map [0]/[0] -- 0/0 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] state: Reset advmap 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] new interval newup [0] newacting [0] 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] state: should restart peering, calling start_peering_interval again 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=18 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] set_last_peering_reset 19 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=19 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] Clearing blocked outgoing recovery messages 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [] r=-1 lpr=19 pi=[9,18)/2 crt=12'2 lcod 0'0 unknown NOTIFY m=1 mbc={}] Not blocking outgoing recovery messages 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] start_peering_interval: check_new_interval output: check_new_interval interval(18-18 up [](-1) acting [](-1)) : acting set is too small 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 18/18/18) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] noting past ([9,18] intervals=([14,15] acting 1),([16,17] acting 0,1)) 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] reg_next_scrub pg 1.0 register next scrub, scrub time 2019-04-21 19:43:18.948439, must = 0 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] _on_new_interval checking missing set deletes flag. missing = missing(1 may_include_deletes = 1) 2019-04-20 14:16:34.524 7f4656280700 1 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] start_peering_interval up [] -> [0], acting [] -> [0], acting_primary ? -> 0, up_primary ? -> 0, role -1 -> 0, features acting 4540138292836696063 upacting 4540138292836696063 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] clear_primary_state 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] agent_stop 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] on_change 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] cancel_copy_ops 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] cancel_flush_ops 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] cancel_proxy_ops 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:34.524 7f4656280700 15 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] publish_stats_to_osd 17:12 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] on_change_cleanup 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] on_change 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] exit NotTrimming 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown NOTIFY m=1 mbc={}] enter NotTrimming 2019-04-20 14:16:34.524 7f4656280700 15 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] clear_stats 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] on_role_change 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] cancel_recovery 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] clear_recovery_state 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] handle_activate_map 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] update_heartbeat_peers -> 0 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] take_waiters 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] exit Reset 0.120380 4 0.001096 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] enter Started 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] enter Start 2019-04-20 14:16:34.524 7f4656280700 1 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] state: transitioning to Primary 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] exit Start 0.000037 0 0.000000 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] enter Started/Primary 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] enter Started/Primary/Peering 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] enter Started/Primary/Peering/GetInfo 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] build_prior all_probe 0,1 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] build_prior maybe_rw interval:16, acting: 0,1 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] build_prior prior osd.1 is down 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] build_prior final: probe 0 down 1 blocked_by {} 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] up_thru 0 < same_since 19, must notify monitor 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:34.524 7f4656280700 15 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] publish_stats_to_osd 19:13 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] exit Started/Primary/Peering/GetInfo 0.000178 0 0.000000 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] enter Started/Primary/Peering/GetLog 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] choose_acting all_info osd.0 1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] calc_replicated_acting newest update on osd.0 with 1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] choose_async_recovery_replicated candidates by cost are: 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] choose_async_recovery_replicated result want=[0] async_recovery= 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] acting_recovery_backfill is 0 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] choose_acting want=[0] backfill_targets= async_recovery_targets= 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: leaving GetLog 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] exit Started/Primary/Peering/GetLog 0.000188 0 0.000000 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] enter Started/Primary/Peering/GetMissing 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: still need up_thru update before going active 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] exit Started/Primary/Peering/GetMissing 0.000035 0 0.000000 2019-04-20 14:16:34.524 7f4656280700 5 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] enter Started/Primary/Peering/WaitUpThru 2019-04-20 14:16:34.524 7f4656280700 20 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] handle_activate_map: Not dirtying info: last_persisted is 17 while current is 19 2019-04-20 14:16:34.524 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] do_peering_event: epoch_sent: 19 epoch_requested: 19 NullEvt 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 19 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] handle_advance_map [0]/[0] -- 0/0 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: Peering advmap 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] adjust_need_up_thru now 19, need_up_thru now false 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: Started advmap 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] check_recovery_sources no source osds () went down 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] handle_activate_map 2019-04-20 14:16:35.544 7f4656280700 7 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: handle ActMap primary 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.544 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] publish_stats_to_osd 19:14 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] take_waiters 2019-04-20 14:16:35.544 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] exit Started/Primary/Peering/WaitUpThru 1.021404 3 0.000838 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] state: Leaving Peering 2019-04-20 14:16:35.544 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 peering m=1 mbc={}] exit Started/Primary/Peering 1.021973 0 0.000000 2019-04-20 14:16:35.544 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] enter Started/Primary/Active 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=16/17 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] state: In Active, about to call activate 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 lcod 0'0 mlcod 0'0 unknown m=1 mbc={}] activate - not complete, missing(1 may_include_deletes = 1) 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 mbc={}] activate setting up missing_loc from shard 0 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={}] needs_recovery primary has 1 missing 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={}] add_batch_sources_info: adding sources in batch 0 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={255={(0+0)=1}}] build_might_have_unfound 2019-04-20 14:16:35.544 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={255={(0+0)=1}}] build_might_have_unfound: built 1 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={255={(0+0)=1}}] discover_all_missing 1 missing, 1 unfound 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 unknown m=1 u=1 mbc={255={(0+0)=1}}] discover_all_missing skipping down osd.1 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] release_backoffs [1:00000000::::0,MAX) 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] rollforward: entry=12'1 (0'0) modify 1:ad551702:::test:head by client.4139.0:1 2019-04-20 14:12:11.032054 0 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard 0 missing= 1 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard NONE missing= 1 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.544 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:15 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: Activate Finished 2019-04-20 14:16:35.544 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] enter Started/Primary/Active/Activating 2019-04-20 14:16:35.544 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] handle_activate_map: Not dirtying info: last_persisted is 19 while current is 20 2019-04-20 14:16:35.544 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] do_peering_event: epoch_sent: 20 epoch_requested: 20 NullEvt 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _activate_committed 20 peer_activated now 0 last_interval_started 16 last_epoch_started 17 same_interval_since 19 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] all_activated_and_committed 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard 0 missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard NONE missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 16/9 les/c/f 17/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 activating+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] do_peering_event: epoch_sent: 20 epoch_requested: 20 AllReplicasActivated 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] share_pg_info 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard 0 missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard NONE missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.552 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:16 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] check_local 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] needs_recovery primary has 1 missing 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] activate not all replicas are up-to-date, queueing recovery 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard 0 missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard NONE missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.552 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:17 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] hit_set_clear 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] agent_stop 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] do_peering_event: epoch_sent: 20 epoch_requested: 20 DoRecovery 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] exit Started/Primary/Active/Activating 0.006429 3 0.001496 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] enter Started/Primary/Active/WaitLocalRecoveryReserved 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] get_recovery_priority recovery priority for pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] is 180, state is 12583938 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard 0 missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing shard NONE missing= 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.552 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:18 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] do_peering_event: epoch_sent: 20 epoch_requested: 20 LocalRecoveryReserved 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] exit Started/Primary/Active/WaitLocalRecoveryReserved 0.000995 1 0.000756 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] enter Started/Primary/Active/WaitRemoteRecoveryReserved 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.000042 0 0.000000 2019-04-20 14:16:35.552 7f4656280700 5 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovery_wait+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] enter Started/Primary/Active/Recovering 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats ml 1 upset size 1 up 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing based degraded 1 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing based misplaced 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:16:35.552 7f4656280700 20 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:16:35.552 7f4656280700 15 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:19 2019-04-20 14:16:35.552 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] queue_recovery -- queuing 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 20 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] handle_advance_map [0]/[0] -- 0/0 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: Active advmap 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: Started advmap 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] check_recovery_sources no source osds () went down 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] handle_activate_map 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: Active: handling ActMap 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] discover_all_missing 1 missing, 1 unfound 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] discover_all_missing skipping down osd.1 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: Active: kicking snap trim 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] queue_recovery -- already queued 2019-04-20 14:26:31.992 7f4656280700 7 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] state: handle ActMap primary 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats actingset 0 upset 0 acting_recovery_backfill 0 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats acting [0] up [0] 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats shard 0 primary objects 1 missing 1 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats ml 1 upset size 1 up 0 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing based degraded 1 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats missing based misplaced 0 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats degraded 2 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] _update_calc_stats misplaced 0 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd reporting purged_snaps [] 2019-04-20 14:26:31.992 7f4656280700 15 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] publish_stats_to_osd 20:20 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] take_waiters 2019-04-20 14:26:31.992 7f4656280700 20 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] handle_activate_map: Not dirtying info: last_persisted is 20 while current is 21 2019-04-20 14:26:31.992 7f4656280700 10 osd.0 pg_epoch: 21 pg[1.0( v 12'2 lc 0'0 (0'0,12'2] local-lis/les=19/20 n=1 ec=9/9 lis/c 19/9 les/c/f 20/10/0 19/19/19) [0] r=0 lpr=19 pi=[9,19)/2 crt=12'2 mlcod 0'0 active+recovering+undersized+degraded m=1 u=1 mbc={255={(0+0)=1}}] do_peering_event: epoch_sent: 21 epoch_requested: 21 NullEvt