Ceph

When a Ceph cluster is busy, it is possible that one or more OSDs become too busy handling replication OPs from other primaries that they can not service OPs from clients. Some sort of feedback mechanism probably should be implemented to help other primary OSDs understand that I/O from their clients should be throttled so that the replica OSDs can also service client I/O.

This issue can be replicated by generating a large amount of client traffic with large queue depths in a cluster an slow messages for only a few I/O will show up for 30+ seconds. A lower number of PGs can also help expose this issue.