Project

General

Profile

Bug #48153

collection_list_legacy: pg inconsistent

Added by Alexander Kazansky 4 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Scrub/Repair
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
nautilus,octopus
Regression:
Yes
Severity:
1 - critical
Reviewed:

Description

hello ppl.

i have problem with osd on nau 14.2.12 and 14.2.13

after update some osd on 14.2.12 or 14.2.13 started happening error inconsistent pg.

ceph pg ls inconsistent

PG     OBJECTS DEGRADED MISPLACED UNFOUND BYTES        OMAP_BYTES* OMAP_KEYS* LOG  STATE                     SINCE VERSION        REPORTED        UP                   ACTING               SCRUB_STAMP                DEEP_SCRUB_STAMP
21.4c1  373274        0         0       0 146034187553           0          0 3055 active+clean+inconsistent    6h 318913'2325690 318913:16123496 [206,430,418,68]p206 [206,430,418,68]p206 2020-11-09 05:44:01.436817 2020-11-09 05:44:01.436817
21.b3d  372613        0         0       0 146754784380           0          0 3089 active+clean+inconsistent   13h 318913'2285226 318913:14445734 [425,334,224,65]p425 [425,334,224,65]p425 2020-11-08 23:15:36.748113 2020-11-08 23:15:36.748113

rados list-inconsistent-objet show what some osds missed replica


"shards": [
    {
        "osd": 68,
        "primary": false,
        "errors": [],
        "size": 181625,
        "omap_digest": "0xffffffff",
        "data_digest": "0x5b4976ac" 
    },
    {
        "osd": 206,
        "primary": true,
        "errors": [],
        "size": 181625,
        "omap_digest": "0xffffffff",
        "data_digest": "0x5b4976ac" 
    },
    {
        "osd": 418,
        "primary": false,
        "errors": [],
        "size": 181625,
        "omap_digest": "0xffffffff",
        "data_digest": "0x5b4976ac" 
    },
    {
        "osd": 430,
        "primary": false,
        "errors": [
            "missing" 
        ]
    }
]

missing replica happens only on osd with 14.2.12 or 14.2.13, on 14.2.11 all ok.


ceph tell osd.430 version
{
    "version": "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)" 
}

downgrade to 14.2.11 also resolve this problem.


Related issues

Copied to RADOS - Backport #48243: octopus: collection_list_legacy: pg inconsistent Resolved
Copied to RADOS - Backport #48244: nautilus: collection_list_legacy: pg inconsistent Resolved

History

#1 Updated by Mykola Golub 4 months ago

Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.

#2 Updated by Mykola Golub 4 months ago

missing replica happens only on osd with 14.2.12 or 14.2.13, on 14.2.11 all ok.

Can I assume that in [206,430,418,68] acting set the osd 430 was on 14.2.12 (or 14.2.13) and all others were on 14.2.11?

#3 Updated by Alexander Kazansky 4 months ago

Mykola Golub wrote:

Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.

Mykola, in current time i have 14.2.11 and 14.2.13

   "osd": {
        "ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)": 423,
        "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 9
    },

rados list-inconsistent-obj 21.4c1 -f json-pretty

{
    "epoch": 318747,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 911562
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 3898983617,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "184508'913352",
                "prior_version": "184487'911562",
                "last_reqid": "osd.206.0:346179",
                "user_version": 911562,
                "size": 181625,
                "mtime": "2019-11-19 15:42:53.934165",
                "local_mtime": "2019-11-19 15:42:53.938977",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0x5b4976ac",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 68,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 206,
                    "primary": true,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 418,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 430,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

rados list-inconsistent-obj 21.725 -f json-pretty

{
    "epoch": 318458,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 470625
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 2330335013,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "73111'509645",
                "prior_version": "72894'470625",
                "last_reqid": "osd.95.0:2798768",
                "user_version": 470625,
                "size": 4194304,
                "mtime": "2019-07-18 03:47:05.424479",
                "local_mtime": "2019-07-18 03:47:05.490979",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0xca716c01",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 24,
                    "primary": false,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 145,
                    "primary": false,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 247,
                    "primary": true,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 428,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

rados list-inconsistent-obj 21.b3d -f json-pretty


{
    "epoch": 318769,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 858487
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 2025421629,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "158482'867618",
                "prior_version": "154486'858487",
                "last_reqid": "osd.219.0:394603",
                "user_version": 858487,
                "size": 169493,
                "mtime": "2019-10-03 14:19:45.275571",
                "local_mtime": "2019-10-03 14:19:45.279188",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0xc13e3929",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 65,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 224,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 334,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 425,
                    "primary": true,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

as i see all affected obj have cyrilic symbols :/

#4 Updated by Mykola Golub 4 months ago

Alexander, I am building a cluster to reproduce the issue.

Meantime, could you please increase debug_osd level on the osd 430 (which I assume is upgraded to 14.2.13) to 10:

ceph tell osd.430 injectargs '--debug-osd=10'

and schedule the pg 21.4c1 deep-scrub:
ceph pg deep-scrub 21.4c1

And after it is run, grep the osd log for "collection_list". I am interested to see if "collection_list" or "collection_list_legacy" is called in your case.

#5 Updated by Mykola Golub 4 months ago

ceph tell osd.430 injectargs '--debug-osd=10'

sorry, it should be '--debug-bluestore=10'.

And if you see "collection_list_legacy" in the log, then I am interested to see more details about object names you have so I could use them to try to reproduce the issue. Taking your example for pg 21.4c1:

            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 3898983617,
...
                {
                    "osd": 418,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 430,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]

It would be helpful to have the object listing from a non-upgraded (418) osd and the upgraded osd 430, stopping the osd and running the command:

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-$id --pgid 21.4c1 --op list > list.$id.txt

If this is erasure coded pool then you need to add the shard to the pgid: 21.4c1s1 for osd 430, and 21.4c1s3 for the osd 418 I believe.

#6 Updated by Mykola Golub 4 months ago

  • Status changed from New to In Progress
  • Assignee set to Mykola Golub

And if you see "collection_list_legacy" in the log, then I am interested to see more details about object names you have so I could use them to try to reproduce the issue.

Never mind, I have reproduced the issue with my data and I think I know the root cause:

`BlueStore::collection_list_legacy` function from the new osd, which is expected to behave exactly as the old old collection_list function (prior 4.2.12), may produce a different result. In `BlueStore::_collection_list`, if "end" object is specified, when checking if the end object is reached we compare the current object with the end, which is correct, but is not correct for the "legacy" case, because the old `BlueStore::_collection_list` is comparing keys, and may produce a longer list (including the end object). I am working on the fix.

Alexander, thank you for reporting this. I think I don't need more info from you. Still if you want to make sure your case is the same what I have reproduced, the easiest way probably would be to rerun the deep-scrub with debug_bluestore=30 set on osds and provide the logs from the upgraded osd and one not upgraded osd, so I could compare `BlueStore::_collection_list` results.

#7 Updated by Mykola Golub 4 months ago

And just as a note. The problem is observed only when the scrub is run for a pg that have both old and new versions of osds in its active set (because the bluestore collection_list behaves a bit differently on different versions).

If you are sure you can upgrade all osds quickly enough (I suppose upgrade the osd binary on all nodes, and then restart), you can just disable the scrub before the upgrade, upgrade all osds and enable it back.

#8 Updated by Mykola Golub 4 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 37999

#9 Updated by Konstantin Shalygin 4 months ago

  • Subject changed from pg inconsistent to collection_list_legacy: pg inconsistent
  • Category set to Scrub/Repair

#10 Updated by Neha Ojha 4 months ago

  • Backport set to nautilus,octopus

#11 Updated by Mykola Golub 3 months ago

  • Status changed from Fix Under Review to Pending Backport

#12 Updated by Mykola Golub 3 months ago

  • Copied to Backport #48243: octopus: collection_list_legacy: pg inconsistent added

#13 Updated by Mykola Golub 3 months ago

  • Copied to Backport #48244: nautilus: collection_list_legacy: pg inconsistent added

#14 Updated by Nathan Cutler about 1 month ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

#15 Updated by Konstantin Shalygin about 1 month ago

  • Target version changed from v14.2.13 to v16.0.0
  • Affected Versions v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.14, v15.2.1, v15.2.2, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8 added

Also available in: Atom PDF