Exceeding osd nearfull ratio causes write throttle.
We noticed a 20x write performance reduction on our CEPHFS cluster shortly after one of our OSDs exceeded the near-full ratio.
Historically (pre nautilus), the "nearfull" warning never had any operational impact, so it took us a while to narrow down the actual cause.
Would be worth adding to user documentation: "If we are near [nearfull] ENOSPC, write synchronously." and the implications on client performance.
Appears to be related to the following commit: