Bug #18533
Updated by Dan Mick over 7 years ago
Twice, twice, I've noticed this sort of error pop on the long-running cluster: one of the metadata objects reports one replica has a different omap_digest: <pre> list-inconsistent-obj 1.3c metadata { "epoch": 771290, "inconsistents": [ { "object": { "name": "607.00000000", "nspace": "", "locator": "", "snap": "head", "version": 8962591 }, "errors": [ "omap_digest_mismatch" ], "union_shard_errors": [], "selected_object_info": "1:3ed09add:::607.00000000:head(769238'8962591 mds.0.95185:3872723 dirty|omap|data_digest s 0 uv 8962591 dd ffffffff alloc_hint [0 0 0])", "shards": [ { "osd": 31, "errors": [], "size": 0, "omap_digest": "0xa99faf1c", "data_digest": "0xffffffff" }, { "osd": 48, "errors": [], "size": 0, "omap_digest": "0x0b59d114", "data_digest": "0xffffffff" }, { "osd": 61, "errors": [], "size": 0, "omap_digest": "0xa99faf1c", "data_digest": "0xffffffff" }, { "osd": 69, "errors": [], "size": 0, "omap_digest": "0xa99faf1c", "data_digest": "0xffffffff" } ] } ] } </pre> This object's omap has one key, 'parent', whose value is: <pre> 00000000 05 04 38 00 00 00 07 06 00 00 00 00 00 00 01 00 |..8.............| 00000010 00 00 02 02 1a 00 00 00 00 01 00 00 00 00 00 00 |................| 00000020 06 00 00 00 73 74 72 61 79 37 47 4d 0e 00 00 00 |....stray7GM....| 00000030 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |..............| 0000003e </pre> On the differing osd, the value is: <pre> ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-48 '["1.3c",{"oid":"607.00000000","key":"","snapid":-2,"hash":3143175036,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _parent | hd 00000000 05 04 38 00 00 00 07 06 00 00 00 00 00 00 01 00 |..8.............| 00000010 00 00 02 02 1a 00 00 00 00 01 00 00 00 00 00 00 |................| 00000020 06 00 00 00 73 74 72 61 79 37 47 4d 0e 00 00 00 |....stray7GM....| 00000030 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |..............| 0000003e </pre> which looks correct to me, but for some reason the omap_digest was calculated differently during the scrub, maybe?.... I've left the object as is.