Ceph » RADOS

[RFE] Stop OSD's removal if the OSD's are part of inactive PGs

Description of problem:

[RFE] Stop OSD's removal if the OSD's are part of inactive PGs.

This kind of OSDs can be taken down and out state but we should stop removing it from OSDmap and throw a big warning that these OSDs are part of inactive PGs and we can hit data loss if they will clean up the OSD store we may require RADOS PGs(objects) from these down and out OSDs.

and if still, the user wants to remove them then they can remove them with the option "--yes-i-really-mean-it".

This will help us to not have unfound objects and incomplete PGs very frequently because of users remove a couple of OSD's very frequently without waiting for PGs to go active+clean and this caused down, inactive, incomplete PGs and very frequent unfound objects.

and this OSD removal stop feature should only get activated if the OSD is part of any inactive PG not in the general use case where all PGs are active+clean and the cluster has finished backfilling the data and these OSD's are just waiting for removal.

Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.y
Jewel
hammer

Downstream feature - https://bugzilla.redhat.com/show_bug.cgi?id=1496571