Ceph » RADOS

We had a situation where due to hardware issues our network was getting lossy for one OSD node, without the Ceph monitoring/heartbeating detecting this. The loss came with a twist: the loss rate would rise with packet size. With only a few bytes payload loss was hardly noticeable, but loss rate would rise quasi-proportionally up to 60-80% with payload sizes up to 7kB (we're using jumboframes with max. MTU 8k).

With those high loss rates most reads from OSDs would basically stall, but still MONs would happily report OSDs as up/in. From my reading of docs/code (my C++ is weak++ sadly) I can see why the heartbeating wouldn't catch this - it's sending relatively small MOSDPing messages on separate channels to detect network partitions; those would have passed largely unimpeded.

Still, the pathological state of those OSDs could have been detected by the MON (packets queueing up, tcp timeouts). From the user POV it would have been hugely desirable to just mark those OSDs as down, or at least warn about unresponsive OSDs.