Ceph » devops

I don't think we want to do this. The problem is that if we plug in a new OSD that has the same ID as the previous one, the system will expect it to have the data that the previous one had.
We can fix that by removing the previous OSD via the monitors, but then on re-add the OSD isn't allowed to select its own ID — the monitors just hand out the first available one.

Greg Farnum wrote:

I don't think we want to do this. The problem is that if we plug in a new OSD that has the same ID as the previous one, the system will expect it to have the data that the previous one had.
We can fix that by removing the previous OSD via the monitors, but then on re-add the OSD isn't allowed to select its own ID — the monitors just hand out the first available one.

Well.. the OSDs will query the existing id to see if it might have data, but they will happily move on if/when it doesn't (just as they would if the osd id didn't exist at all). This isn't any different from what happens now, since ids are reused.

Ah, right. I was thinking we could get into badness over that disagreement, but of course everything checks the real state before making any changes based on presumed state.

Still, this does mean that if you manage to recover the disk and plug it in elsewhere you can't use its contents in recovery (if, say, it has some otherwise-lost objects on it). I think we should take this feature request as another indication that we need to finish divorcing OSD names and IDs, rather than as meaning we should make re-use of IDs easier...

One path forward is to make the 'osd create <uuid>' optionally take the osd id we would like, and use it if available (or fail). Then ceph-disk-prepare can do the same.

Just changing whoami isn't enough.. the osd uuid needs to map to the id in the osdmap. Unless we are re-using an existing uuid with the same id... but just wedging it into ceph-disk-prepare without mon confirmation risks making a 'broken' disk with a id and uuid that don't match and therefore can't be used.