Ceph » RADOS

2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [ RUN      ] LibRadosListPP.EnumerateObjectsPP
2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-237-gef10e30/rpm/el7/BUILD/ceph-12.1.1-237-gef10e30/src/test/librados/list.cc:1150: Failure
2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Expected: n_objects
2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: 16
2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: To be equal to: saw_obj.size()
2017-07-20T20:38:27.053 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: 95
2017-07-20T20:38:27.055 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-237-gef10e30/rpm/el7/BUILD/ceph-12.1.1-237-gef10e30/src/test/librados/TestCase.cc:115: Failure
2017-07-20T20:38:27.055 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Expected: 0
2017-07-20T20:38:27.055 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: To be equal to: ioctx.remove(it->get_oid())
2017-07-20T20:38:27.055 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: -2
2017-07-20T20:38:27.055 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [  FAILED  ] LibRadosListPP.EnumerateObjectsPP (3153 ms)
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [ RUN      ] LibRadosListPP.EnumerateObjectsSplitPP
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-237-gef10e30/rpm/el7/BUILD/ceph-12.1.1-237-gef10e30/src/test/librados/list.cc:1206: Failure
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Expected: n_objects
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: 16
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: To be equal to: saw_obj.size()
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: 92
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-237-gef10e30/rpm/el7/BUILD/ceph-12.1.1-237-gef10e30/src/test/librados/TestCase.cc:115: Failure
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Expected: 0
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: To be equal to: ioctx.remove(it->get_oid())
2017-07-20T20:38:27.056 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: -2
2017-07-20T20:38:27.057 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [  FAILED  ] LibRadosListPP.EnumerateObjectsSplitPP (3567 ms)
2017-07-20T20:38:27.057 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [ RUN      ] LibRadosListPP.EnumerateObjectsFilterPP
2017-07-20T20:38:27.057 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-237-gef10e30/rpm/el7/BUILD/ceph-12.1.1-237-gef10e30/src/test/librados/TestCase.cc:115: Failure
2017-07-20T20:38:27.057 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Expected: 0
2017-07-20T20:38:27.057 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: To be equal to: ioctx.remove(it->get_oid())
2017-07-20T20:38:27.058 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list:       Which is: -2
2017-07-20T20:38:27.058 INFO:tasks.workunit.client.0.smithi023.stdout:                 api_list: [  FAILED  ] LibRadosListPP.EnumerateObjectsFilterPP (1850 ms)

and osd log indicates the delete op triggered the missing delete to happen:

2017-07-20 20:38:17.388806 2e585700 10 osd.3 pg_epoch: 224 pg[101.4( v 224'77 lc 128'1 (0'0,224'77] local-lis/les=214/215 n=3 ec=126/126 lis/c 214/126 les/c/f 215/127/0 199/214/214) [3,4] r=0 lpr=214 pi=[126,214)/1 crt=224'77 mlcod 128'1 active+recovery_wait+degraded m=27] start_recovery_op 101:28883081:::19:head
2017-07-20 20:38:17.389190 2e585700 10 osd.3 224 start_recovery_op pg[101.4( v 224'77 lc 128'1 (0'0,224'77] local-lis/les=214/215 n=3 ec=126/126 lis/c 214/126 les/c/f 215/127/0 199/214/214) [3,4] r=0 lpr=214 pi=[126,214)/1 rops=1 crt=224'77 mlcod 128'1 active+recovery_wait+degraded m=27] 101:28883081:::19:head (1/3 rops)
2017-07-20 20:38:17.389544 2e585700 20 osd.3 pg_epoch: 224 pg[101.4( v 224'77 lc 128'1 (0'0,224'77] local-lis/les=214/215 n=3 ec=126/126 lis/c 214/126 les/c/f 215/127/0 199/214/214) [3,4] r=0 lpr=214 pi=[126,214)/1 rops=1 crt=224'77 mlcod 128'1 active+recovery_wait+degraded m=27] remove_missing_object 101:28883081:::19:head 151'34

/a/yuriw-2017-07-20_19:48:38-rados-wip-yuri-testing3_2017_7_21-distro-basic-smithi/1425376