Bug #17588
closedrbd-mirror: disabling mirroring with option '--force' makes RBD-images unaccessible
0%
Description
Hi!
I have two CEPH clusters and just tried to use a [rbd-mirroring]1. One cluster operates with ceph version 10.2.2, second with ceph version 10.2.3. I have installed rbd-mirror daemons on both sides, they work fine, I can mirror rbd-images, promote them, resync etc. Then I wanted to test failover capability: what is happening, if the image is mirroring, but the primary cluster is going down? I promote my non-primary image with option --force. The image got primary status in both clusters (maybe it is actually not, but I am saying, how it is shown). Then I disabled the mirroring (with --force) on anyone nodes and tried to remove this image, but I could not.
root@test-hoster-kvm-01:~# rbd rm rbdkvm_sata/test-rbd-mirroring-root
2016-09-28 13:23:54.019417 7fee29ffb700 -1 librbd::journal::StandardPolicy: local image not promoted
2016-09-28 13:23:54.019430 7fee29ffb700 -1 librbd::exclusive_lock::AcquireRequest: failed to allocate journal tag: (1) Operation not permitted
2016-09-28 13:23:54.044298 7fee29ffb700 -1 librbd::ExclusiveLock: failed to acquire exclusive lock:(1) Operation not permitted
2016-09-28 13:23:54.044395 7fee4e368d40 -1 librbd: cannot obtain exclusive lock - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
I see error message: `error: image still has watchers`. But there are no watchers:
root@test-hoster-kvm-01:~# rbd status rbdkvm_sata/test-rbd-mirroring-root
Watchers: none
More info can be seen below:
root@test-hoster-kvm-01:~# rbd info rbdkvm_sata/test-rbd-mirroring-root
rbd image 'test-rbd-mirroring-root':
size 4096 MB in 1024 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.f63036b8b4567
format: 2
features: layering, exclusive-lock, journaling
flags:
journal: f63036b8b4567
mirroring state: disabled
I can not remove any locks from image, because rbd does not see them:
root@test-mon-01:~# rbd lock list rbdkvm_sata/test-rbd-mirroring-root root@test-mon-01:~#
When I try to enable mirroring again and promote the image, I got an error:
root@test-hoster-kvm-buffer-01a:~# rbd mirror image enable rbdkvm_sata/test-rbd-mirroring-root 2016-09-30 17:23:19.798621 7fb16e6f2d40 -1 librbd: cannot enable mirroring: last journal tag not owned by local cluster`
Because I use CEPH with libvirt, I can't now work with my libvirt rbd-pool (create images, refresh pool):
root@test-mon-01:~# virsh pool-refresh rbdkvm_sata error: Failed to refresh pool rbdkvm_sata error: Requested operation is not valid: storage pool 'rbdkvm_sata' is not active root@test-mon-01:~# virsh pool-start rbdkvm_sata error: Failed to start pool rbdkvm_sata error: failed to open the RBD image 'test-rbd-mirroring-root': Function not implemented
Only `systemctl libvirtd restart` helps me to bring my pool back in active mode.
lsof and ps show that there are no processes on monitors or hosters using this image. But on one of storages the image is open by ceph-osd. Here is an example:
ms_pipe_w 13602 990366 ceph 146u REG 8,97 16 805434978 /var/lib/ceph/osd/ceph-9/current/4.147_head/rbd\uid.test-rbd-mirroring-root__head_E0701D47__4`
I was testing this many times. And this is my conclusion: if you disable image mirroring on non-primary node (this means using option --force), every time your data becomes your data becomes unaccessible. And you can not remove this image from ceph. I know, it is not the best practice, but I think, this is very bad, that we can lose access to disk data because of inputed command, which should just disable the mirroring (at least I could not regain access to my data, maybe you know methods).
I tried mount such image to a virtual server as secondary disk, but then correct starting of the VS is impossible.
So, I have next questions:
1) Is this the bug of rbd-mirror?
2) Maybe I am wrong and did something fool, but how should `rbd mirror image disable {pool/image} --force` work?
3) Can I repair such images (no locks, no watchers, not mapped, used only by storage)?
4) How can I remove such images from ceph? Just remove whithout repairing.
My Debian versions are 8.5 and 8.6.
Please tell me if you need more information or something of the written you do not understand.