Ceph

Test description: rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}

http://158.69.78.47:8081/ubuntu-2016-07-27_20:28:31-rados-jewel---basic-openstack/124

Problems start in the scrub_test phase here: 2016-07-27T21:54:26.593 INFO:tasks.scrub_test:messing with PG 0.17 on osd 0

Here's what's supposed to happen:

log.info('messing with PG %s on osd %d' % (pg, osd))
test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path)
test_repair_bad_omap(ctx, manager, pg, osd, obj_name)
test_list_inconsistent_obj(ctx, manager, osd_remote, pg, acting, osd,
                           obj_name, obj_path)
log.info('test successful!')

The test_list_inconsistent_obj fails. The PG is supposed to show one inconsistent object, but it doesn't:

2016-07-27T21:59:29.685 INFO:teuthology.orchestra.run.target158069095058:Running: 'rados list-inconsistent-obj 0.17 --format=json'
2016-07-27T21:59:29.772 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761396 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-07-27T21:59:29.773 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.761561 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-07-27T21:59:29.775 INFO:teuthology.orchestra.run.target158069095058.stderr:2016-07-27 21:59:29.763813 7fca059fda40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-07-27T21:59:29.786 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 66, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 45, in run_one_task
    return fn(**kwargs)
  File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 397, in task
    obj_name, obj_path)
  File "/home/ubuntu/src/ceph-qa-suite_wip-opensuse-jewel/tasks/scrub_test.py", line 328, in test_list_inconsistent_obj
    assert len(objs) == 1
AssertionError

It's unclear from the output what the actual number of inconsistent objects is.

Added a pprint on "objs" to see what's in it, found the answer: objs==None

Failure happens every time.

Reproducer: cw -v ceph-qa-suite --upload --suite rados --filter="rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/many.yaml tasks/scrub_test.yaml}" --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-jewel --ceph jewel distros/all/ubuntu_14.04.yaml