Bug #24792
ceph-volume fails to zap dmcrypt device on bluestore
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The device cas prepared like this:
[root@ceph-osd0 /]# ceph-volume lvm prepare --data /dev/sda --dmcrypt Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph-authtool --gen-print-key Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea618678-b6d7-489a-9180-a7b67c3b69a0 Running command: /usr/sbin/vgcreate --force --yes ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2 /dev/sda stdout: Physical volume "/dev/sda" successfully created. stdout: Volume group "ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2" successfully created Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2 stdout: Logical volume "osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0" created. Running command: /bin/ceph-authtool --gen-print-key Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 Running command: /usr/sbin/cryptsetup --key-file - luksOpen /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0 Running command: /bin/chown -R ceph:ceph /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 Running command: /bin/ln -s /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 /var/lib/ceph/osd/ceph-0/block Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap stderr: got monmap epoch 1 Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQCWbD9bKrs6JxAA9WGlfT56+6EqogRIL1tDAQ== stdout: creating /var/lib/ceph/osd/ceph-0/keyring added entity osd.0 auth auth(auid = 18446744073709551615 key=AQCWbD9bKrs6JxAA9WGlfT56+6EqogRIL1tDAQ== with 0 caps) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/ Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid ea618678-b6d7-489a-9180-a7b67c3b69a0 --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/sda
See the list:
[root@ceph-osd0 /]# ceph-volume lvm list /dev/sda ====== osd.0 ======= [block] /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 type block osd id 0 cluster fsid 0a9374ec-2998-47df-9a7f-d030e5b9c261 cluster name ceph osd fsid ea618678-b6d7-489a-9180-a7b67c3b69a0 encrypted 1 cephx lockbox secret AQCbbD9btcZ7NxAA1X+fydCuBVkyZZ6yZfymUw== block uuid BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 block device /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 vdo 0 crush device class None devices /dev/sda
First attempt to zap:
[root@ceph-osd0 /]# umount /var/lib/ceph/osd/ceph-0 [root@ceph-osd0 /]# ceph-volume lvm zap /dev/sda --> Zapping: /dev/sda --> Closing encrypted path /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 Running command: /usr/sbin/cryptsetup remove /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 Running command: /usr/sbin/wipefs --all /dev/sda stderr: wipefs: error: /dev/sda: probing initialization failed: Device or resource busy --> RuntimeError: command returned non-zero exit status: 1 [root@ceph-osd0 /]# ps axuf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ceph 1616 0.1 3.8 845844 19276 ? Ssl 12:31 0:05 ceph-osd -i 1 --setuser ceph --setgroup disk root 1229 0.0 0.3 11820 1692 pts/4 Ss 12:04 0:00 bash root 3328 0.0 0.3 51704 1696 pts/4 R+ 13:25 0:00 \_ ps axuf root 836 0.0 0.3 11820 1544 pts/3 Ss+ 08:53 0:00 bash root 5 0.0 0.1 11820 556 pts/1 Ss+ Jul05 0:00 bash root 1 0.0 0.0 4360 24 ? Ss Jul05 0:00 sleep 365d
As we can see OSD.0 was not running, so there might be a timing issue in the removal sequence.
Second run fails too:
[root@ceph-osd0 /]# ceph-volume lvm zap /dev/sda --> Zapping: /dev/sda --> Closing encrypted path /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 Running command: /usr/sbin/cryptsetup remove /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 stderr: Device /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 is not active. --> RuntimeError: command returned non-zero exit status: 4
This means it's not able to resume the operation.
I wonder if this doesn't share so commonality with http://tracker.ceph.com/issues/24504
History
#1 Updated by Alfredo Deza over 5 years ago
- Status changed from New to 4
Now that #24504 is completed, can you check if this is still an issue for you?
#2 Updated by Alfredo Deza over 5 years ago
Ping?
#3 Updated by Sébastien Han over 5 years ago
Sorry I don't have much time to test this again, I'll let you know once I resume my work.
Thanks for the patches.
#4 Updated by Alfredo Deza over 5 years ago
- Status changed from 4 to Resolved