Bug #48153
closedcollection_list_legacy: pg inconsistent
0%
Description
hello ppl.
i have problem with osd on nau 14.2.12 and 14.2.13after update some osd on 14.2.12 or 14.2.13 started happening error inconsistent pg.
ceph pg ls inconsistent
PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
21.4c1 373274 0 0 0 146034187553 0 0 3055 active+clean+inconsistent 6h 318913'2325690 318913:16123496 [206,430,418,68]p206 [206,430,418,68]p206 2020-11-09 05:44:01.436817 2020-11-09 05:44:01.436817
21.b3d 372613 0 0 0 146754784380 0 0 3089 active+clean+inconsistent 13h 318913'2285226 318913:14445734 [425,334,224,65]p425 [425,334,224,65]p425 2020-11-08 23:15:36.748113 2020-11-08 23:15:36.748113
rados list-inconsistent-objet show what some osds missed replica
"shards": [
{
"osd": 68,
"primary": false,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 206,
"primary": true,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 418,
"primary": false,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 430,
"primary": false,
"errors": [
"missing"
]
}
]
missing replica happens only on osd with 14.2.12 or 14.2.13, on 14.2.11 all ok.
ceph tell osd.430 version
{
"version": "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)"
}
downgrade to 14.2.11 also resolve this problem.
Updated by Mykola Golub over 3 years ago
Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.
Updated by Mykola Golub over 3 years ago
missing replica happens only on osd with 14.2.12 or 14.2.13, on 14.2.11 all ok.
Can I assume that in [206,430,418,68] acting set the osd 430 was on 14.2.12 (or 14.2.13) and all others were on 14.2.11?
Updated by Alexander Kazansky over 3 years ago
Mykola Golub wrote:
Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.
Mykola, in current time i have 14.2.11 and 14.2.13
"osd": {
"ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)": 423,
"ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 9
},
rados list-inconsistent-obj 21.4c1 -f json-pretty
{
"epoch": 318747,
"inconsistents": [
{
"object": {
"name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
"nspace": "",
"locator": "",
"snap": "head",
"version": 911562
},
"errors": [],
"union_shard_errors": [
"missing"
],
"selected_object_info": {
"oid": {
"oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
"key": "",
"snapid": -2,
"hash": 3898983617,
"max": 0,
"pool": 21,
"namespace": ""
},
"version": "184508'913352",
"prior_version": "184487'911562",
"last_reqid": "osd.206.0:346179",
"user_version": 911562,
"size": 181625,
"mtime": "2019-11-19 15:42:53.934165",
"local_mtime": "2019-11-19 15:42:53.938977",
"lost": 0,
"flags": [
"dirty",
"data_digest",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0x5b4976ac",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 68,
"primary": false,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 206,
"primary": true,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 418,
"primary": false,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 430,
"primary": false,
"errors": [
"missing"
]
}
]
}
]
}
rados list-inconsistent-obj 21.725 -f json-pretty
{
"epoch": 318458,
"inconsistents": [
{
"object": {
"name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
"nspace": "",
"locator": "",
"snap": "head",
"version": 470625
},
"errors": [],
"union_shard_errors": [
"missing"
],
"selected_object_info": {
"oid": {
"oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
"key": "",
"snapid": -2,
"hash": 2330335013,
"max": 0,
"pool": 21,
"namespace": ""
},
"version": "73111'509645",
"prior_version": "72894'470625",
"last_reqid": "osd.95.0:2798768",
"user_version": 470625,
"size": 4194304,
"mtime": "2019-07-18 03:47:05.424479",
"local_mtime": "2019-07-18 03:47:05.490979",
"lost": 0,
"flags": [
"dirty",
"data_digest",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xca716c01",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 24,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xca716c01"
},
{
"osd": 145,
"primary": false,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xca716c01"
},
{
"osd": 247,
"primary": true,
"errors": [],
"size": 4194304,
"omap_digest": "0xffffffff",
"data_digest": "0xca716c01"
},
{
"osd": 428,
"primary": false,
"errors": [
"missing"
]
}
]
}
]
}
rados list-inconsistent-obj 21.b3d -f json-pretty
{
"epoch": 318769,
"inconsistents": [
{
"object": {
"name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
"nspace": "",
"locator": "",
"snap": "head",
"version": 858487
},
"errors": [],
"union_shard_errors": [
"missing"
],
"selected_object_info": {
"oid": {
"oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
"key": "",
"snapid": -2,
"hash": 2025421629,
"max": 0,
"pool": 21,
"namespace": ""
},
"version": "158482'867618",
"prior_version": "154486'858487",
"last_reqid": "osd.219.0:394603",
"user_version": 858487,
"size": 169493,
"mtime": "2019-10-03 14:19:45.275571",
"local_mtime": "2019-10-03 14:19:45.279188",
"lost": 0,
"flags": [
"dirty",
"data_digest",
"omap_digest"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xc13e3929",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
},
"shards": [
{
"osd": 65,
"primary": false,
"errors": [],
"size": 169493,
"omap_digest": "0xffffffff",
"data_digest": "0xc13e3929"
},
{
"osd": 224,
"primary": false,
"errors": [],
"size": 169493,
"omap_digest": "0xffffffff",
"data_digest": "0xc13e3929"
},
{
"osd": 334,
"primary": false,
"errors": [],
"size": 169493,
"omap_digest": "0xffffffff",
"data_digest": "0xc13e3929"
},
{
"osd": 425,
"primary": true,
"errors": [
"missing"
]
}
]
}
]
}
as i see all affected obj have cyrilic symbols :/
Updated by Mykola Golub over 3 years ago
Alexander, I am building a cluster to reproduce the issue.
Meantime, could you please increase debug_osd level on the osd 430 (which I assume is upgraded to 14.2.13) to 10:
ceph tell osd.430 injectargs '--debug-osd=10'
and schedule the pg 21.4c1 deep-scrub:
ceph pg deep-scrub 21.4c1
And after it is run, grep the osd log for "collection_list". I am interested to see if "collection_list" or "collection_list_legacy" is called in your case.
Updated by Mykola Golub over 3 years ago
ceph tell osd.430 injectargs '--debug-osd=10'
sorry, it should be '--debug-bluestore=10'.
And if you see "collection_list_legacy" in the log, then I am interested to see more details about object names you have so I could use them to try to reproduce the issue. Taking your example for pg 21.4c1:
"selected_object_info": {
"oid": {
"oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
"key": "",
"snapid": -2,
"hash": 3898983617,
...
{
"osd": 418,
"primary": false,
"errors": [],
"size": 181625,
"omap_digest": "0xffffffff",
"data_digest": "0x5b4976ac"
},
{
"osd": 430,
"primary": false,
"errors": [
"missing"
]
}
]
It would be helpful to have the object listing from a non-upgraded (418) osd and the upgraded osd 430, stopping the osd and running the command:
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-$id --pgid 21.4c1 --op list > list.$id.txt
If this is erasure coded pool then you need to add the shard to the pgid: 21.4c1s1 for osd 430, and 21.4c1s3 for the osd 418 I believe.
Updated by Mykola Golub over 3 years ago
- Status changed from New to In Progress
- Assignee set to Mykola Golub
And if you see "collection_list_legacy" in the log, then I am interested to see more details about object names you have so I could use them to try to reproduce the issue.
Never mind, I have reproduced the issue with my data and I think I know the root cause:
`BlueStore::collection_list_legacy` function from the new osd, which is expected to behave exactly as the old old collection_list function (prior 4.2.12), may produce a different result. In `BlueStore::_collection_list`, if "end" object is specified, when checking if the end object is reached we compare the current object with the end, which is correct, but is not correct for the "legacy" case, because the old `BlueStore::_collection_list` is comparing keys, and may produce a longer list (including the end object). I am working on the fix.
Alexander, thank you for reporting this. I think I don't need more info from you. Still if you want to make sure your case is the same what I have reproduced, the easiest way probably would be to rerun the deep-scrub with debug_bluestore=30 set on osds and provide the logs from the upgraded osd and one not upgraded osd, so I could compare `BlueStore::_collection_list` results.
Updated by Mykola Golub over 3 years ago
And just as a note. The problem is observed only when the scrub is run for a pg that have both old and new versions of osds in its active set (because the bluestore collection_list behaves a bit differently on different versions).
If you are sure you can upgrade all osds quickly enough (I suppose upgrade the osd binary on all nodes, and then restart), you can just disable the scrub before the upgrade, upgrade all osds and enable it back.
Updated by Mykola Golub over 3 years ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 37999
Updated by Konstantin Shalygin over 3 years ago
- Subject changed from pg inconsistent to collection_list_legacy: pg inconsistent
- Category set to Scrub/Repair
Updated by Mykola Golub over 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Mykola Golub over 3 years ago
- Copied to Backport #48243: octopus: collection_list_legacy: pg inconsistent added
Updated by Mykola Golub over 3 years ago
- Copied to Backport #48244: nautilus: collection_list_legacy: pg inconsistent added
Updated by Nathan Cutler over 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Updated by Konstantin Shalygin over 3 years ago
- Target version changed from v14.2.13 to v16.0.0
- Affected Versions v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.14, v15.2.1, v15.2.2, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8 added