Ceph » rbd

Thanks for looking into this Christopher. You are right, this is a 100% sequential workload, just filling a volume with random data. I don't think caching has to do with the throughput drop, or at least not caching of the data that is being written to the volume: the amount of data we write is orders of magnitude larger than the amount of RAM of either the client or the server machines, so the cache should be full after only a few seconds. However, we observe a high throughput during dozens of minutes or even hours before the drop occurs. And that high throughput (100 - 130 MB/s) is actually more or less in line with what can be expected of an SSD drive; it is probably even higher during the first few seconds, until the caches are full.

Without knowing anything about the internals of Ceph or RBD, I was thinking that maybe the Ceph client needs to maintain a kind of mapping of all the blocks that constitute the volume, and if so, the bigger the volume becomes, the larger that mapping needs to be, which would explain why the drop occurs once the volume reaches a certain size depending on available RAM on the client (assuming that if the mapping cannot fit in client RAM it has to be retrieved from the server, thus killing the throughput). Of course that is just wild speculation on my part.

In order to rule out if client side caching is a factor can you do a few tests?

Run tests without client side caching:
1. Write to device using kernel with direct=1.
fio --name=write --filename=/dev/rbd0 --ioengine=libaio --iodepth=1 --rw=write --bs=1M --direct=1 --size=800G --group_reporting

2. Write to image in user space using librbd with direct=1
fio --name=write --ioengine=rbd --clientname=admin --pool=rbd --rbdname=fio_test --invalidate=0 --rw=write --bs=1M --iodepth=1 --direct=1 --size=800G --group_reporting

Run test with client side caching:
3. Run test 1 and 2 with direct=0

:facepalm: the issue has nothing to do with Ceph and everything to do with smaller EC2 instances having burstable network bandwidth, proportional to the number of vCPUs, which is incidentally also proportional to the available RAM. I was not aware of that. Such instances are allowed to use the maximum bandwidth for some time, after which they are throttled to a much lower bandwidth, hence the drop in throughput I was observing. Sorry for the noise.

For reference, this is documented here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose-instances.html#general-purpose-network-performance