Actions
Bug #27363
open'rbd rm' does not clean tiered pool completly
Added by Fyodor Ustinov over 5 years ago. Updated over 5 years ago.
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
mimic (13.2.1)
linux kernel: 4.18.3-1.el7.elrepo.x86_64
ceph osd crush rule create-replicated hddreplrule default rack hdd
ceph osd crush rule create-replicated ssdreplrule default rack ssd
ceph osd pool create rbd 256 256 replicated hddreplrule
ceph osd pool set rbd size 3
ceph osd pool set rbd min_size 2
ceph osd pool create rbdtier 256 256 replicated ssdreplrule
ceph osd pool set rbdtier size 3
ceph osd pool set rbdtier min_size 2
ceph osd tier add-cache rbd rbdtier 549755813888
ceph osd tier cache-mode rbdtier writeback
rbd create --size 16G rbdtest
rbd feature disable rbdtest object-map fast-diff deep-flatten
rbd device map rbdtest
dd if=/dev/zero of=/dev/rbd0 bs=16M oflag=direct
dd if=/dev/zero of=/dev/rbd0 bs=16M
rbd device unmap rbdtest
rbd rm rbdtest
Removing image: 100% complete...done.
rbd trash purge
rbd: nothing to remove
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 11 5.2 GiB 0.01 35 TiB 1328
rbdtier 12 0 B 0 1.7 TiB 1330
rados -p rbd ls
rbd_data.15196b8b4567.00000000000009c5
rbd_data.15196b8b4567.00000000000008ed
rbd_data.15196b8b4567.0000000000000dd6
rbd_data.15196b8b4567.0000000000000501
[...]
rados -p rbdtier ls
rbd_data.15196b8b4567.00000000000009c5
rbd_data.15196b8b4567.00000000000008ed
rbd_data.15196b8b4567.0000000000000dd6
rbd_data.15196b8b4567.0000000000000501
[...]
Updated by Jason Dillaman over 5 years ago
- Project changed from rbd to RADOS
Moving this to the core team. This appears to be an issue w/ the cache tier. In my test, after removing the image, all the data objects were deleted but the main image objects "remained" as ghost whiteout objects in the cache tier (note that these objects are guaranteed to not have snapshots unlike the data objects):
$ rados -p rbd ls rbd_id.rbdtest rbd_directory rbd_header.104b452037b0 rbd_object_map.104b452037b0 rbd_info $ rados -p rbdtier ls rbd_id.rbdtest rbd_directory rbd_header.104b452037b0 rbd_object_map.104b452037b0 rbd_mirroring $ rados -p rbd rm rbd_id.rbdtest error removing rbdtier>rbd_id.rbdtest: (2) No such file or directory $ rados -p rbdtier rm rbd_id.rbdtest error removing rbdtier>rbd_id.rbdtest: (2) No such file or directory
From the logs:
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] do_op: op osd_op(client.14130.0:4124 2.4 2:30966a08:::rbd_id.rbdtest:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try e32) v8
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] op_has_sufficient_caps session=0x55d4fffba2c0 pool=2 (rbdtier ) pool_app_metadata={} need_read_cap=0 need_write_cap=1 classes=[] -> yes
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] do_op osd_op(client.14130.0:4124 2.4 2:30966a08:::rbd_id.rbdtest:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try e32) v8 may_write -> write-ordered flags ondisk+write+known_if_redirected+full_try
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] get_object_context: obc NOT found in cache: 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] populate_obc_watchers 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] PrimaryLogPG::check_blacklisted_obc_watchers for obc 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] get_object_context: creating obc from disk: 0x55d5005c1080
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] get_object_context: 0x55d5005c1080 2:30966a08:::rbd_id.rbdtest:head rwstate(none n=0 w=0) oi: 2:30966a08:::rbd_id.rbdtest:head(31'2 osd.1.0:3 data_digest s 15 uv 2 dd 696098b8 alloc_hint [0 0 0]) exists: 1 ssc: 0x55d4ff389080 snapset: 0=[]:{}
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] find_object_context 2:30966a08:::rbd_id.rbdtest:head @head oi=2:30966a08:::rbd_id.rbdtest:head(31'2 osd.1.0:3 data_digest s 15 uv 2 dd 696098b8 alloc_hint [0 0 0])
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] agent_choose_mode flush_mode: idle evict_mode: idle num_objects: 2 num_bytes: 15 num_objects_dirty: 0 num_objects_omap: 1 num_dirty: 0 num_user_objects: 2 num_user_bytes: 615 num_overhead_bytes: 600 pool.info.target_max_bytes: 549755813888 pool.info.target_max_objects: 0
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] agent_choose_mode dirty 0 full 0
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] do_op obc obc(2:30966a08:::rbd_id.rbdtest:head rwstate(write n=1 w=0))
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] execute_ctx 0x55d504423800
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] execute_ctx 2:30966a08:::rbd_id.rbdtest:head [delete] ov 31'2 av 32'3 snapc 0=[] snapset 0=[]:{}
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] do_osd_op 2:30966a08:::rbd_id.rbdtest:head [delete]
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] do_osd_op delete
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] _delete_oid 2:30966a08:::rbd_id.rbdtest:head whiteout=1 no_whiteout=0 try_no_whiteout=0
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] _delete_oid setting whiteout on 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] make_writeable 2:30966a08:::rbd_id.rbdtest:head snapset=0=[]:{} snapc=0=[]
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] setting DIRTY flag
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] make_writeable 2:30966a08:::rbd_id.rbdtest:head done, snapset=0=[]:{}
2018-09-13 09:25:32.494 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] finish_ctx 2:30966a08:::rbd_id.rbdtest:head 0x55d504423800 op modify
2018-09-13 09:25:32.494 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] set mtime to 2018-09-13 09:25:32.495211
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] final snapset 0=[]:{} in 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.495 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] zeroing write result code 0
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] calc_trim_to limit = 31'2
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] new_repop rep_tid 3772 on osd_op(client.14130.0:4124 2.4 2:30966a08:::rbd_id.rbdtest:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try e32) v8
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] new_repop: repgather(0x55d4ff316300 0'0 rep_tid=3772 committed?=0 r=0)
2018-09-13 09:25:32.495 7fc7146b9700 7 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] issue_repop rep_tid 3772 o 2:30966a08:::rbd_id.rbdtest:head
2018-09-13 09:25:32.495 7fc7146b9700 20 osd.1 32 share_map_peer 0x55d4ff18d100 already has epoch 32
2018-09-13 09:25:32.495 7fc7146b9700 20 osd.1 32 share_map_peer 0x55d4ff18a000 already has epoch 32
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 31'2 (0'0,31'2] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] append_log log((0'0,31'2], crt=31'2) [32'3 (31'2) modify 2:30966a08:::rbd_id.rbdtest:head by client.14130.0:4124 2018-09-13 09:25:32.495211 0]
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 lua=31'2 crt=31'2 lcod 31'1 mlcod 31'1 active+clean] add_log_entry 32'3 (31'2) modify 2:30966a08:::rbd_id.rbdtest:head by client.14130.0:4124 2018-09-13 09:25:32.495211 0
2018-09-13 09:25:32.495 7fc7146b9700 20 trim 32'3 (31'2) modify 2:30966a08:::rbd_id.rbdtest:head by client.14130.0:4124 2018-09-13 09:25:32.495211 0
2018-09-13 09:25:32.495 7fc7146b9700 20 earliest_dup_version = 0
2018-09-13 09:25:32.495 7fc7146b9700 20 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 lua=31'2 crt=32'3 lcod 31'1 mlcod 31'1 active+clean] rollforward: entry=32'3 (31'2) modify 2:30966a08:::rbd_id.rbdtest:head by client.14130.0:4124 2018-09-13 09:25:32.495211 0
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 lua=31'2 crt=32'3 lcod 31'1 mlcod 31'1 active+clean] append_log approx pg log length = 3
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 lua=31'2 crt=32'3 lcod 31'1 mlcod 31'1 active+clean] append_log transaction_applied = 1
2018-09-13 09:25:32.495 7fc7146b9700 10 trim proposed trim_to = 0'0
2018-09-13 09:25:32.495 7fc7146b9700 5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 32'3, trimmed: , trimmed_dups: , clear_divergent_priors: 0
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 lua=31'2 crt=32'3 lcod 31'1 mlcod 31'1 active+clean] op_applied version 32'3
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'3 (0'0,32'3] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 luod=31'2 crt=32'3 lcod 31'1 mlcod 31'1 active+clean] eval_repop repgather(0x55d4ff316300 32'3 rep_tid=3772 committed?=0 r=0)
2018-09-13 09:25:32.495 7fc7146b9700 10 osd.1 32 dequeue_op 0x55d505355dc0 finish
...
2018-09-13 09:40:10.143 7fc7186c1700 20 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] do_op: op osd_op(client.14142.0:1 2.4 2:30966a08:::rbd_id.rbdtest:head [stat] snapc 0=[] ondisk+read+known_if_redirected e32) v8
2018-09-13 09:40:10.143 7fc7186c1700 20 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] op_has_sufficient_caps session=0x55d516126dc0 pool=2 (rbdtier ) pool_app_metadata={} need_read_cap=1 need_write_cap=0 classes=[] -> yes
2018-09-13 09:40:10.143 7fc7186c1700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] do_op osd_op(client.14142.0:1 2.4 2:30966a08:::rbd_id.rbdtest:head [stat] snapc 0=[] ondisk+read+known_if_redirected e32) v8 may_read -> read-ordered flags ondisk+read+known_if_redirected
2018-09-13 09:40:10.144 7fc7186c1700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] get_object_context: found obc in cache: 0x55d5005c1080
2018-09-13 09:40:10.144 7fc7186c1700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] get_object_context: 0x55d5005c1080 2:30966a08:::rbd_id.rbdtest:head rwstate(none n=0 w=0) oi: 2:30966a08:::rbd_id.rbdtest:head(32'3 client.14130.0:4124 whiteout|dirty s 0 uv 3 alloc_hint [0 0 0]) exists: 1 ssc: 0x55d4ff389080 snapset: 0=[]:{}
2018-09-13 09:40:10.144 7fc7186c1700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] find_object_context 2:30966a08:::rbd_id.rbdtest:head @head oi=2:30966a08:::rbd_id.rbdtest:head(32'3 client.14130.0:4124 whiteout|dirty s 0 uv 3 alloc_hint [0 0 0])
2018-09-13 09:40:10.144 7fc7186c1700 10 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] agent_choose_mode flush_mode: idle evict_mode: idle num_objects: 2 num_bytes: 0 num_objects_dirty: 2 num_objects_omap: 1 num_dirty: 2 num_user_objects: 2 num_user_bytes: 600 num_overhead_bytes: 600 pool.info.target_max_bytes: 549755813888 pool.info.target_max_objects: 0
2018-09-13 09:40:10.144 7fc7186c1700 20 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] agent_choose_mode dirty 0 full 0
2018-09-13 09:40:10.144 7fc7186c1700 20 osd.1 pg_epoch: 32 pg[2.4( v 32'4 (0'0,32'4] local-lis/les=28/29 n=2 ec=21/21 lis/c 28/28 les/c/f 29/29/0 21/28/21) [1,0,2] r=0 lpr=28 crt=32'4 lcod 32'3 mlcod 32'3 active+clean] do_op obc obc(2:30966a08:::rbd_id.rbdtest:head rwstate(read n=1 w=0))
2018-09-13 09:40:10.144 7fc7186c1700 10 osd.1 32 dequeue_op 0x55d503dd7340 finish
Actions