Ceph » RADOS

Workaround for this issue is to set osd_async_recovery_min_cost to a very large value.

# ceph config set osd osd_async_recovery_min_cost 1099511627776

Notes from Sam : The async recovery cost is the number of pg log entries behind on the replica + the number of missing objects. The osd_target_pg_log_entries_per_osd is 30000, so an OSD with a single PG could have 30000 entries. The osd_async_recovery_min_cost is a 64-bit integer so set it to 2^40 (1<<40) i.e to 1099511627776 value which we cannot hit.

Hello. Just FYI, this fixes a very nasty issue in my EC setup.
Here are some details.

The EC setup and crush rules are defined to have:
3 racks
2 hosts per rack
12 disks per host

EC configuration 7+5
Crush rule picks 1 rack, 2 hosts and 2 disks so 4 chunks of data in each rack.

Now here is the funny thing I end up thanks to this bug.

1. Start some IO
2. Shutdown OSDs from rack A (pg's are active+undersized)
3. Start OSDs from rack A (pg's are active+undersized)
4. Again Shutdown OSDs from rack A (some PG's are down)

So in theory when rack is down 4 chunks are unavailable but 8 are still present - all PGs should remain active.

Now even more weird, continuing the case described above:

4a. Stop IO
5. Disable recovery/rebalance to make sure no chunks are recovered
6. Start OSDs from rack A (again pg's are active+undersized)
7. Again Shutdown OSDs from rack A (some PG's are down again but much less than in step 4, e.g. 50 instead of 300)
8. Start OSDs from rack A (again pg's are active+undersized)
9. Again Shutdown OSDs from rack A (some PG's are down again but much less than in step 7, e.g. 5 instead of 50)
10. Start OSDs from rack A (again pg's are active+undersized)
11. Again Shutdown OSDs from rack A (all pg's are active+undersized)

Further rack restarts won't cause PG down any more, unless there is some IO.

So my guess here is that async recovery kicks in when rack A goes up for the first time, it messes up with the acting set and as a result when rack A goes down again some unfortunate PGs end up in down state.

I retested everything a couple of times with `osd_async_recovery_min_cost 1099511627776` on reef - no more down PGs.

Thank you very very much for the fix.

Hello again.

Apparently I got a tiny little bit too excited.

I tested the case described above with 16.2.15 and unfortunately the problem still exists.
However if I disable async recovery (osd_async_recovery_min_cost 1099511627776) then cluster works as desired, all PGs states are as expected (active + undersized, never down)

I'd appreciate any tips on how to narrow this bug down.