Bug #16841
Updated by Nathan Cutler almost 8 years ago
Test description: rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml} http://158.69.78.47:8081/ubuntu-2016-07-27_20:28:31-rados-jewel---basic-openstack/124 Problems start I ran the rados suite against jewel in the scrub_test phase here: <code>2016-07-27T21:54:26.593 INFO:tasks.scrub_test:messing with PG 0.17 on osd 0</code> Here's what's supposed to happen: <pre> log.info('messing with PG %s on osd %d' % (pg, osd)) test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path) test_repair_bad_omap(ctx, manager, pg, osd, obj_name) test_list_inconsistent_obj(ctx, manager, osd_remote, pg, acting, osd, obj_name, obj_path) log.info('test successful!') </pre> The <code>test_list_inconsistent_obj</code> fails. The PG is supposed to show one inconsistent object, but it doesn't: <pre> 2016-07-27T21:59:29.685 INFO:teuthology.orchestra.run.target158069095058:Running: 'rados list-inconsistent-obj 0.17 --format=json' 2016-07-27T21:59:29.772 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761396 7fca059fda40 -1 WARNING: the following dangerous OpenStack and experimental features are enabled: * 2016-07-27T21:59:29.773 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761561 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-07-27T21:59:29.775 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.763813 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-07-27T21:59:29.786 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 66, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 45, in run_one_task return fn(**kwargs) File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 397, in task obj_name, obj_path) File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 328, in test_list_inconsistent_obj assert len(objs) == 1 AssertionError </pre> got more failures than expected: It's unclear from the output what the actual number of inconsistent objects is. http://158.69.78.47:8081/ubuntu-2016-07-27_20:28:31-rados-jewel---basic-openstack/ Added a pprint on "objs" to see what's in it, found the answer: objs==None Failure happens every time. Reproducer: <code>cw -v ceph-qa-suite --upload --suite rados --filter="rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}" --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-jewel --ceph jewel distros/all/ubuntu_14.04.yaml</code>