Project

General

Profile

Bug #18533

Updated by Dan Mick almost 2 years ago

Twice, twice, I've noticed this sort of error pop on the long-running cluster: one of the metadata objects reports one replica has a different omap_digest:

<pre>
list-inconsistent-obj 1.3c metadata
{
"epoch": 771290,
"inconsistents": [
{
"object": {
"name": "607.00000000",
"nspace": "",
"locator": "",
"snap": "head",
"version": 8962591
},
"errors": [
"omap_digest_mismatch"
],
"union_shard_errors": [],
"selected_object_info": "1:3ed09add:::607.00000000:head(769238'8962591 mds.0.95185:3872723 dirty|omap|data_digest s 0 uv 8962591 dd ffffffff alloc_hint [0 0 0])",
"shards": [
{
"osd": 31,
"errors": [],
"size": 0,
"omap_digest": "0xa99faf1c",
"data_digest": "0xffffffff"
},
{
"osd": 48,
"errors": [],
"size": 0,
"omap_digest": "0x0b59d114",
"data_digest": "0xffffffff"
},
{
"osd": 61,
"errors": [],
"size": 0,
"omap_digest": "0xa99faf1c",
"data_digest": "0xffffffff"
},
{
"osd": 69,
"errors": [],
"size": 0,
"omap_digest": "0xa99faf1c",
"data_digest": "0xffffffff"
}
]
}
]
}
</pre>

This object's omap has one key, 'parent', whose value is:
<pre>
00000000 05 04 38 00 00 00 07 06 00 00 00 00 00 00 01 00 |..8.............|
00000010 00 00 02 02 1a 00 00 00 00 01 00 00 00 00 00 00 |................|
00000020 06 00 00 00 73 74 72 61 79 37 47 4d 0e 00 00 00 |....stray7GM....|
00000030 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |..............|
0000003e
</pre>

On the differing osd, the value is:

<pre>
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-48 '["1.3c",{"oid":"607.00000000","key":"","snapid":-2,"hash":3143175036,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _parent | hd

00000000 05 04 38 00 00 00 07 06 00 00 00 00 00 00 01 00 |..8.............|
00000010 00 00 02 02 1a 00 00 00 00 01 00 00 00 00 00 00 |................|
00000020 06 00 00 00 73 74 72 61 79 37 47 4d 0e 00 00 00 |....stray7GM....|
00000030 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |..............|
0000003e

</pre>

which looks correct to me, but for some reason the omap_digest was calculated differently during the scrub, maybe?....

I've left the object as is.

Back