Ceph

The problem is that when you are removing a single osd from a single host, almost all of that osd's pgs tend to shuffle over to other osds in the same bucket. Somewhat more other pgs also shuffle from other osds in that bucket over to other buckets (since the bucket size shrank). This tends to mean that the other osds in the same host have a bunch of work to do when an osd is removed from the crush map. With current crush tunables, the situation is considerably better if instead you mark an osd down and out -- the pgs tend to redistribute elsewhere, not to the same bucket. I think the workflow when an osd fails needs to be that the osd is killed and marked out immediately, and asynchronously, the weight is gradually reduced until it becomes 0. At that point it can be removed without impact. To support this, the mon needs a

ceph crush begin-removal <osdid>

This command should fail if <osdid> is up or in. It should cause other commands to treat that osd as if it did not exist. However, it should not allow reuse of the osdid. In the background while the cluster is healthy, the mons should gradually reduce the weight until the weight is 0. At that point, the osd should be removed from the crush map. I don't think it's desirable that the osd be resurrectable after begin-removal is issued.

More definition is required here before anyone actually does any work on it!

We probably also want to generalize the operation to any crush sub-tree.