Ceph » rbd

This happens with rbd-mirror-snapshot-stress-workunit-fast-diff workunit as well:

2023-04-21T13:56:14.592 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:2023-04-21T13:56:14.589+0000 7f3ae5136640 -1 bdev(0x563cea16ae00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device
2023-04-21T13:56:14.593 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:2023-04-21T13:56:14.593+0000 7f3ae5136640 -1 bdev(0x563cea16ae00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device
2023-04-21T13:56:14.599 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:2023-04-21T13:56:14.597+0000 7f3ae5136640 -1 bdev(0x563cea16ae00 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device
2023-04-21T13:56:14.599 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:2023-04-21T13:56:14.597+0000 7f3aeb94f640 -1 bdev(0x563cea16ae00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device)
2023-04-21T13:56:14.600 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f3aeb94f640 time 2023-04-21T13:56:14.602226+0000
2023-04-21T13:56:14.600 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!")
2023-04-21T13:56:14.600 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: ceph version 18.0.0-3518-g35b1d1cd (35b1d1cd2cb020f91e80d1e2aa54c0c472a0d004) reef (dev)
2023-04-21T13:56:14.600 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x563ce5e5d49a]
2023-04-21T13:56:14.600 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x563ce6719b7c]
2023-04-21T13:56:14.601 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: 3: ceph-osd(+0x1581431) [0x563ce671a431]
2023-04-21T13:56:14.601 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f3afa7a7b43]
2023-04-21T13:56:14.601 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f3afa839a00]
2023-04-21T13:56:14.601 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:2023-04-21T13:56:14.597+0000 7f3aeb94f640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f3aeb94f640 time 2023-04-21T13:56:14.602226+0000
2023-04-21T13:56:14.601 INFO:tasks.ceph.cluster2.osd.0.smithi189.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!")

And appears to be persistent (3/3):

https://pulpito.ceph.com/dis-2023-04-20_13:01:32-rbd-wip-dis-testing-distro-default-smithi/7246281
https://pulpito.ceph.com/dis-2023-04-21_07:39:27-rbd-wip-dis-testing-distro-default-smithi/7247480
https://pulpito.ceph.com/dis-2023-04-21_12:49:35-rbd-wip-dis-testing-distro-default-smithi/7247692

@Ilya this looks like a dupe of https://tracker.ceph.com/issues/49138.