Bug #13726
QEMU hangs after creating snapshot and stopping VM
0%
Description
When RBD cache is disabled, taking a snapshot and stopping the VM results in a hung QEMU process. Setting "rbd_non_blocking_aio" to false apparently resolves the issue.
Related issues
Associated revisions
librbd: fixed deadlock while attempting to flush AIO requests
In-flight AIO requests might force a flush if a snapshot was created
out-of-band. The flush completion was previously invoked asynchronously,
potentially via the same thread worker handling the AIO request. This
resulted in the flush operation deadlocking since it can't complete.
Fixes: #13726
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
librbd: fixed deadlock while attempting to flush AIO requests
In-flight AIO requests might force a flush if a snapshot was created
out-of-band. The flush completion was previously invoked asynchronously,
potentially via the same thread worker handling the AIO request. This
resulted in the flush operation deadlocking since it can't complete.
Fixes: #13726
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bfeb90e5fe24347648c72345881fd3d932243c98)
librbd: fixed deadlock while attempting to flush AIO requests
In-flight AIO requests might force a flush if a snapshot was created
out-of-band. The flush completion was previously invoked asynchronously,
potentially via the same thread worker handling the AIO request. This
resulted in the flush operation deadlocking since it can't complete.
Fixes: #13726
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit bfeb90e5fe24347648c72345881fd3d932243c98)
History
#1 Updated by alexandre derumier over 4 years ago
- File ceph.client.log View added
I have attached the client log,
the snapshot create command was :
rbd -p pooltest --image vm-162-disk-1 snap create --snap snap1
#2 Updated by Jason Dillaman over 4 years ago
In-flight AIO read request forces an image refresh due to out-of-band snapshot creation. Detecting the newly created snapshot forces librbd to flush all in-flight ops. If there are no in-flight ops to flush, it will enqueue a completion on the thread pool. However, the thread pool will be blocked handling the AIO request waiting for the flush to complete.
The synchronous ImageCtx::flush_async_operations() method cannot use async callbacks.
#3 Updated by Jason Dillaman over 4 years ago
- Backport set to hammer,infernalis
#4 Updated by Jason Dillaman over 4 years ago
- Status changed from New to In Progress
- Assignee set to Jason Dillaman
#5 Updated by Jason Dillaman over 4 years ago
- Status changed from In Progress to Fix Under Review
jewel PR: https://github.com/ceph/ceph/pull/6508
#6 Updated by alexandre derumier over 4 years ago
I confirm this is fixed with this PR.
(tested on infernalis)
#7 Updated by Loic Dachary over 4 years ago
- Status changed from Fix Under Review to Pending Backport
#8 Updated by Loic Dachary over 4 years ago
- Copied to Backport #13755: QEMU hangs after creating snapshot and stopping VM added
#9 Updated by Loic Dachary over 4 years ago
- Copied to Backport #13756: QEMU hangs after creating snapshot and stopping VM added
#10 Updated by Jason Dillaman over 4 years ago
- Status changed from Pending Backport to Resolved
#11 Updated by Nathan Cutler about 4 years ago
- Related to Bug #14988: QEMU VM hangs talking to RBD via librbd added