Ceph » RADOS

Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.

missing replica happens only on osd with 14.2.12 or 14.2.13, on 14.2.11 all ok.

Can I assume that in [206,430,418,68] acting set the osd 430 was on 14.2.12 (or 14.2.13) and all others were on 14.2.11?

Mykola Golub wrote:

Alexander, did you have osds of different versions in your acting set? If you did, what exactly they were?
Also is it possible for you to show the full output for `rados list-inconsistent-object`? The object names are important here.

Mykola, in current time i have 14.2.11 and 14.2.13

   "osd": {
        "ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)": 423,
        "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 9
    },

rados list-inconsistent-obj 21.4c1 -f json-pretty

{
    "epoch": 318747,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 911562
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 3898983617,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "184508'913352",
                "prior_version": "184487'911562",
                "last_reqid": "osd.206.0:346179",
                "user_version": 911562,
                "size": 181625,
                "mtime": "2019-11-19 15:42:53.934165",
                "local_mtime": "2019-11-19 15:42:53.938977",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0x5b4976ac",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 68,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 206,
                    "primary": true,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 418,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 430,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

rados list-inconsistent-obj 21.725 -f json-pretty

{
    "epoch": 318458,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 470625
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/фото_2_tLgunsT.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 2330335013,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "73111'509645",
                "prior_version": "72894'470625",
                "last_reqid": "osd.95.0:2798768",
                "user_version": 470625,
                "size": 4194304,
                "mtime": "2019-07-18 03:47:05.424479",
                "local_mtime": "2019-07-18 03:47:05.490979",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0xca716c01",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 24,
                    "primary": false,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 145,
                    "primary": false,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 247,
                    "primary": true,
                    "errors": [],
                    "size": 4194304,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xca716c01" 
                },
                {
                    "osd": 428,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

rados list-inconsistent-obj 21.b3d -f json-pretty

{
    "epoch": 318769,
    "inconsistents": [
        {
            "object": {
                "name": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
                "nspace": "",
                "locator": "",
                "snap": "head",
                "version": 858487
            },
            "errors": [],
            "union_shard_errors": [
                "missing" 
            ],
            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/гагарина_6_wYd4qwi.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 2025421629,
                    "max": 0,
                    "pool": 21,
                    "namespace": "" 
                },
                "version": "158482'867618",
                "prior_version": "154486'858487",
                "last_reqid": "osd.219.0:394603",
                "user_version": 858487,
                "size": 169493,
                "mtime": "2019-10-03 14:19:45.275571",
                "local_mtime": "2019-10-03 14:19:45.279188",
                "lost": 0,
                "flags": [
                    "dirty",
                    "data_digest",
                    "omap_digest" 
                ],
                "truncate_seq": 0,
                "truncate_size": 0,
                "data_digest": "0xc13e3929",
                "omap_digest": "0xffffffff",
                "expected_object_size": 0,
                "expected_write_size": 0,
                "alloc_hint_flags": 0,
                "manifest": {
                    "type": 0
                },
                "watchers": {}
            },
            "shards": [
                {
                    "osd": 65,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 224,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 334,
                    "primary": false,
                    "errors": [],
                    "size": 169493,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0xc13e3929" 
                },
                {
                    "osd": 425,
                    "primary": true,
                    "errors": [
                        "missing" 
                    ]
                }
            ]
        }
    ]
}

as i see all affected obj have cyrilic symbols :/

Alexander, I am building a cluster to reproduce the issue.

Meantime, could you please increase debug_osd level on the osd 430 (which I assume is upgraded to 14.2.13) to 10:

ceph tell osd.430 injectargs '--debug-osd=10'

ceph pg deep-scrub 21.4c1

ceph tell osd.430 injectargs '--debug-osd=10'

sorry, it should be '--debug-bluestore=10'.

And if you see "collection_list_legacy" in the log, then I am interested to see more details about object names you have so I could use them to try to reproduce the issue. Taking your example for pg 21.4c1:

            "selected_object_info": {
                "oid": {
                    "oid": "4e9fcbd7-89b9-4423-b798-383dae23f6b0.21514923.2_petition/petition_response/None/Кисляково-Базыкино.jpg",
                    "key": "",
                    "snapid": -2,
                    "hash": 3898983617,
...
                {
                    "osd": 418,
                    "primary": false,
                    "errors": [],
                    "size": 181625,
                    "omap_digest": "0xffffffff",
                    "data_digest": "0x5b4976ac" 
                },
                {
                    "osd": 430,
                    "primary": false,
                    "errors": [
                        "missing" 
                    ]
                }
            ]

It would be helpful to have the object listing from a non-upgraded (418) osd and the upgraded osd 430, stopping the osd and running the command:

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-$id --pgid 21.4c1 --op list > list.$id.txt

And just as a note. The problem is observed only when the scrub is run for a pg that have both old and new versions of osds in its active set (because the bluestore collection_list behaves a bit differently on different versions).

If you are sure you can upgrade all osds quickly enough (I suppose upgrade the osd binary on all nodes, and then restart), you can just disable the scrub before the upgrade, upgrade all osds and enable it back.