Bug #16841
closedrados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml} broken in jewel
0%
Description
Test description: rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}
http://158.69.78.47:8081/ubuntu-2016-07-27_20:28:31-rados-jewel---basic-openstack/124
Problems start in the scrub_test phase here: 2016-07-27T21:54:26.593 INFO:tasks.scrub_test:messing with PG 0.17 on osd 0
Here's what's supposed to happen:
log.info('messing with PG %s on osd %d' % (pg, osd)) test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path) test_repair_bad_omap(ctx, manager, pg, osd, obj_name) test_list_inconsistent_obj(ctx, manager, osd_remote, pg, acting, osd, obj_name, obj_path) log.info('test successful!')
The test_list_inconsistent_obj
fails. The PG is supposed to show one inconsistent object, but it doesn't:
2016-07-27T21:59:29.685 INFO:teuthology.orchestra.run.target158069095058:Running: 'rados list-inconsistent-obj 0.17 --format=json' 2016-07-27T21:59:29.772 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761396 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-07-27T21:59:29.773 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761561 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-07-27T21:59:29.775 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.763813 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-07-27T21:59:29.786 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 66, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 45, in run_one_task return fn(**kwargs) File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 397, in task obj_name, obj_path) File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 328, in test_list_inconsistent_obj assert len(objs) == 1 AssertionError
It's unclear from the output what the actual number of inconsistent objects is.
Added a pprint on "objs" to see what's in it, found the answer: objs==None
Failure happens every time.
Reproducer: cw -v ceph-qa-suite --upload --suite rados --filter="rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}" --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-jewel --ceph jewel distros/all/ubuntu_14.04.yaml
Updated by Nathan Cutler over 7 years ago
- Subject changed from rados+jewel: excessive failures in OpenStack to rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml} broken in jewel
- Description updated (diff)
Updated by Nathan Cutler over 7 years ago
Trying to reproduce in Sepia with
teuthology-suite --verbose --suite rados --filter="rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}" --suite-branch jewel --ceph jewel --machine-type smithi distros/all/ubuntu_14.04.yaml
running http://pulpito.ceph.com/smithfarm-2016-08-03_04:21:27-rados-jewel---basic-smithi/
Updated by Loïc Dachary over 7 years ago
- Project changed from 23 to Ceph
- Status changed from New to Duplicate