Ceph » rbd

When appending to a journal object the number of appends sent in one rados operation is not limited and we may hit osd_max_write_size limit. It is more likely to observe when using very large rbd cache.

It looks like it is much harder to trigger on the master branch, but on luminous it is easily reproduced with this scenario:

rbd create test -s 100G
rbd feature enable test journaling
rbd-nbd --rbd_cache_max_dirty=1549551462 --rbd_cache_size=1610612736 --rbd_cache_target_dirty=1449551462 map test
mkfs.xfs -f /dev/nbd0
mount /dev/nbd0 /mnt
cd /mnt
fio --name=writefile --filesize=1G --bs=4M --nrfiles=16 --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=16 --ioengine=libaio

Eventually it fails with:

  librbd::Journal: 0x7f52440037e0 handle_io_event_safe: failed to commit IO event: (90) Message too long