Backport #23700
Updated by Nathan Cutler about 6 years ago
https://github.com/ceph/ceph/pull/21407 Hi, Got a bug at one of my OSD's, see a snippet below. Full logfile also attached. I've tested with "dev_aio_max_queue_depth = 4096" as some gave as a workaround, but it didn't work. I'll be glad to assist with more information when needed. 2018-03-06 15:50:35.045499 7f34b893ce00 1 freelist init 2018-03-06 15:50:35.145876 7f34b893ce00 1 bluestore(/var/lib/ceph/osd/ceph-6) _open_alloc opening allocation metadata 2018-03-06 15:51:41.962675 7f34b893ce00 1 bluestore(/var/lib/ceph/osd/ceph-6) _open_alloc loaded 718 G in 1331688 extents 2018-03-06 15:51:51.199593 7f34b893ce00 -1 bdev(0x563e2b634fc0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 16 2018-03-06 15:51:51.199801 7f34b893ce00 -1 bdev(0x563e2b634fc0 /var/lib/ceph/osd/ceph-6/block) aio submit got (11) Resource temporarily unavailable 2018-03-06 15:51:51.207950 7f34b893ce00 -1 /build/ceph-12.2.4/src/os/bluestore/KernelDevice.cc: In function 'virtual void KernelDevice::aio_submit(IOContext*)' thread 7f34b893ce00 time 2018-03-06 15:51:51.199884 /build/ceph-12.2.4/src/os/bluestore/KernelDevice.cc: 539: FAILED assert(r == 0) ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x563e21d76872] 2: (KernelDevice::aio_submit(IOContext*)+0x6ed) [0x563e21d1088d] 3: (BlueStore::_deferred_submit_unlock(BlueStore::OpSequencer*)+0x779) [0x563e21bc5e49] 4: (BlueStore::deferred_try_submit()+0x7b1) [0x563e21bc6d71] 5: (BlueStore::_osr_drain_all()+0x31d) [0x563e21bd0e0d] 6: (BlueStore::_deferred_replay()+0x4ad) [0x563e21c2b40d] 7: (BlueStore::_mount(bool)+0x437) [0x563e21c39b77] 8: (OSD::init()+0x3e2) [0x563e217a0ed2] 9: (main()+0x2f07) [0x563e216b1f17] 10: (__libc_start_main()+0xf0) [0x7f34b5da5830] 11: (_start()+0x29) [0x563e2173d6b9] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.