Ceph

A "postpone PG removal" flag would be useful if you know beforehand that the OSD will be removed after it's PGs have been moved to other OSDs. Instead of spending work on removing PGs, just skip this altogether, as it's a waste of resources. Even more, it can cause impact (slow ops because of "log_latency_fn slow operation observed for _txc_committed_kv", or even OSD reboots / PG degradation because of osd_op_tp thread timeouts).

What are the use cases? When re-provisioning of OSDs is needed, i.e.

- convert OSDs to another type of OSD (bluestore > CRIMSON)
convert to encrypted OSDs
- RocksDB resharding (instead of "fsck'ing")
- hardware replacements

and probably many more.

If this turns out not feasible to implement in current OSD implementation it would make sense to add this support to next gen OSDs (CRIMSON and beyond)