Ceph

When osds are marked out, the recovery ops per primary throttling works fine, because the backfill targets tend to be spread all over.

When you mark an osd in, there tends to be a single backfill target. The larger the cluster, the larger the number of primary osds who are trying to send it data.

I think we need to use a reservation system similar to what we do for scrubbing so that there are configurable number of backfills targetting a single OSD.