Ceph » RADOS

Checking acting.size() < pool.info.min_size is wrong. During recovery acting == up. So if active.size() < pool.info.min_size, the result of the recovery will still be too low to allow I/O. This would be a very uncommon case plus it doesn't help us to prioritize doing that recovery. Also, recovery allows simultaneous operations because even if data hasn't transferred, those operations can be logged on all OSDs. I'm not sure how this interacts with pg log size limiting. If recovery is slow due to large objects and clients send lots of small operations, the count of log entries is going to go up. Will log trimming slow client requests, so the log doesn't grow unbounded?

When recovery is degraded, we don't differentiate whether or not there are at least min_size replicas of every object. In that case we would want a priority boost, but I'm not sure the best way to determine between the 2 cases. Also, as recovery proceeds the degraded will go down, but at the point that example 1 has 100 degraded like example 2 it is really 100 objects on OSD 1 and 50 objects on OSD 0 and OSD 2. So in actual fact 50 objects are still below min_size assuming we push the same object to both OSDs as recovery goes along.

Example 1, size 3 min_size 2 replicated pool
Should get a priority boost because for 100 objects has 200 degraded (below min_size with 100 replicas existing)
PG 1.0 [1, 0, 2] active+clean
OSDs 0 and 2 go down and out
PG 1.0 [1, 3, 4]. active+recovering+degraded

Example 2, size 3 min_size 2 replicated pool
No priority boost because 100 objects has 100 degraded (min_size 2 ok with 200 existing replicas)
PG 1.0 [1, 0, 2] active+clean
OSDs 2 go down and out
PG 1.0 [1, 0, 3]. active+recovering+degraded

David, we should discuss whether we want to backport this all the way to luminous or just to nautilus.