bluestore: aio submission deadlock
- thread a holds deferred_submit_lock, blocks on aio submission (queue is full)
- thread b holds deferred_lock, blocks taking deferred_submit_lock
- aio completion handler blocks on deferred_lock, cannot drain aio queue.
#6 Updated by Sage Weil about 5 years ago
There wsa also an aio submission bug that dropped ios on the floor. it was consistently reproducible with
make ceph_test_objectstore && rm -rf bluestore*test*dir c && CEPH_ARGS="--log-file c --no-log-to-stderr --debug-bluestore 20 --debug-bdev 20 --bdev-debug-aio --bdev-aio-max-queue-depth 16 --bluestore-cache-trim-interval .05" bin/ceph_test_objectstore --gtest_filter=*Syn*/2 --gtest_filter=ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixCompression/2
on an nvme. that bug is also fixed by the pr.
#13 Updated by Bob Bobington about 5 years ago
Since my issue (http://tracker.ceph.com/issues/21314) was marked as a dupe of this and I haven't received a response to the updates on that issue in a week, thought I'd add here as well: The fixes given haven't lead to any improvement for me. I still consistently see the same problems.
I've tried applying this fix as well as adding some of the workarounds suggested but my OSDs still crash with the same messages.