Ceph » RADOS

Thanks Graham,

I'll be taking a look into this. I can confirm I can reproduce the issue locally with the osdmaptool and that I will investigate the cause and update here when I have some findings.

$ bin/osdmaptool --health osdmap.20171207
bin/osdmaptool: osdmap file 'osdmap.20171207'
{
    "OSD_ORPHAN": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "3 osds exist in the crush map but not in the osdmap" 
        },
        "detail": [
            {
                "message": "osd.2 exists in crush map but not in osdmap" 
            },
            {
                "message": "osd.14 exists in crush map but not in osdmap" 
            },
            {
                "message": "osd.19 exists in crush map but not in osdmap" 
            }
        ]
    }
}

So this is happening because entries for "device2", "device14", and "device19" still have entries in the "name_map" section of the crushmap. This appears to indicate they have been only partially removed from the crushmap.

$ osdmaptool --health osdmap.20171207
osdmaptool: osdmap file 'osdmap.20171207'
{
    "OSD_ORPHAN": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "3 osds exist in the crush map but not in the osdmap" 
        },
        "detail": [
            {
                "message": "osd.2 exists in crush map but not in osdmap" 
            },
            {
                "message": "osd.14 exists in crush map but not in osdmap" 
            },
            {
                "message": "osd.19 exists in crush map but not in osdmap" 
            }
        ]
    }
}
$ osdmaptool --export-crush crushmap osdmap.20171207
osdmaptool: osdmap file 'osdmap.20171207'
osdmaptool: exported crush map to crushmap
$ crushtool -i crushmap --remove-item device2 -o crushmap
crushtool removing item device2
$ crushtool -i crushmap --remove-item device14 -o crushmap
crushtool removing item device14
$ crushtool -i crushmap --remove-item device19 -o crushmap
crushtool removing item device19
$ osdmaptool --import-crush crushmap osdmap.20171207
osdmaptool: osdmap file 'osdmap.20171207'
osdmaptool: imported 17117 byte crush map from crushmap
osdmaptool: writing epoch 493339 to osdmap.20171207
$ osdmaptool --health osdmap.20171207
osdmaptool: osdmap file 'osdmap.20171207'
{}

I suspect the solution here is to perform the same removal on the crushmap of your cluster but I'd like to discuss how this might have come about and the best course of action with some of my colleagues and will advise when I've done that.

Interesting! It seems like we probably removed 30 osds from the old retired hardware, so it's curious that just 3 had these traces left behind.

Thanks, and I'll be ready to try whatever you suggest.

Hi Graham,

The consensus is that this was caused by a bug in a previous release which failed to remove the devices in question completely. Therefore the solution should be to get the crushmap, remove the devices as above and reinject the modified crushmap.

That did clean it up, thanks.

It is curious though that if I decompile the crushmap to text, it appears the same both before and after using crushtool to remove the devices.

Brad Hubbard wrote:

Hi Graham,

The consensus is that this was caused by a bug in a previous release which failed to remove the devices in question completely. Therefore the solution should be to get the crushmap, remove the devices as above and reinject the modified crushmap.

Hi Brad, does the bug already been fixed it in Luminous? can you post the PR number?

Hi Jun,

It's not really possible to pinpoint an exact PR at this stage as it's possible there was more than one and it's also possible that this state was arrived at by an unusual sequence of steps, or a sequence involving unusual circumstances or events. Basically, if you can reproduce this state with current tools and can provide the steps you followed to lead to this state we's like to hear about it.