Ceph

A best guess omap_digest_mismatch marked an arbitrary osd as mismatched. The ceph-qa-suite expects the scrub to divine which was the wrong one which it can't.

2016-05-26T01:23:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2016-05-26 08:23:33.865050 7f9abdbd0a40 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-05-26T01:23:33.924 INFO:tasks.scrub_test:inc = {u'object': {u'snap': u'head', u'locator': u'', u'nspace': u'', u'name': u'benchmark_data_smithi042_15995_object1126'}, u'errors': [u'omap_digest_mismatch'], u'shards': [{u'omap_digest': u'0xdf5e79ad', u'errors': [], u'data_digest': u'0x683e2523', u'osd': 0, u'size': 4096}, {u'omap_digest': u'0x3d709c88', u'errors': [u'omap_digest_mismatch'], u'data_digest': u'0x683e2523', u'osd': 5, u'size': 4096}]}
2016-05-26T01:23:33.924 INFO:tasks.scrub_test:shard = {u'omap_digest': u'0xdf5e79ad', u'errors': [], u'data_digest': u'0x683e2523', u'osd': 0, u'size': 4096}
2016-05-26T01:23:33.924 INFO:tasks.scrub_test:attr = omap_digest
2016-05-26T01:23:33.925 INFO:tasks.scrub_test:shard = {u'omap_digest': u'0x3d709c88', u'errors': [u'omap_digest_mismatch'], u'data_digest': u'0x683e2523', u'osd': 5, u'size': 4096}
2016-05-26T01:23:33.925 INFO:tasks.scrub_test:attr = omap_digest
2016-05-26T01:23:33.925 INFO:tasks.scrub_test:shard = {u'omap_digest': u'0xdf5e79ad', u'errors': [], u'data_digest': u'0x683e2523', u'osd': 0, u'size': 4096}
2016-05-26T01:23:33.925 INFO:tasks.scrub_test:err = omap_digest_mismatch
2016-05-26T01:23:33.926 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 66, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 45, in run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/scrub_test.py", line 399, in task
    obj_name, obj_path)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/scrub_test.py", line 337, in test_list_inconsistent_obj
    checker.run(check, inc_obj)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/scrub_test.py", line 230, in run
    func(inc)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/scrub_test.py", line 300, in omap_digest_mismatch
    self._check_errors(inc, 'omap_digest_mismatch')
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/scrub_test.py", line 245, in _check_errors
    "Didn't find '{err}' in errors".format(err=err_name)
AssertionError: Didn't find 'omap_digest_mismatch' in errors

2016-05-26 08:22:35.394854 7fe52121a700 -1 log_channel(cluster) log [ERR] : 0.17 shard 5: soid 0:e85cd6e1:::benchmark_data_smithi042_15995_object1126:head omap_digest 0x3d709c88 != best guess omap_digest 0xdf5e79ad from auth shard 0