Ceph

Say we have a cap of N aio ops, to prevent a stream of small ops resulting in a stream of tiny aio writes to the journal. (Okay, actually, is that actually bad? Maybe not!)

Anyway, assuming it is, and we implement a cap, we can do something like wait a minimum of (bytes_in_flight / max_journal_tput / N) seconds before dispatching the next aio write, which will make each write, on average, (current_throughput / N) bytes. This would avoid dispatching N 4k writes, hitting our max, and waiting until the first completes, but not having enough time to get the next io to the controller before the disk rotates again (since all N writes were small and right next to each other).

Presumably AIO writes can be combined or reordered by the block device/interfaces, right? So having a bunch of them in-flight would be more likely to increase than decrease performance.