Project

General

Profile

Bug #24792

ceph-volume fails to zap dmcrypt device on bluestore

Added by Sébastien Han over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The device cas prepared like this:

[root@ceph-osd0 /]# ceph-volume lvm prepare --data /dev/sda --dmcrypt
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea618678-b6d7-489a-9180-a7b67c3b69a0
Running command: /usr/sbin/vgcreate --force --yes ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2 /dev/sda
 stdout: Physical volume "/dev/sda" successfully created.
 stdout: Volume group "ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2" successfully created
Running command: /usr/sbin/lvcreate --yes -l 100%FREE -n osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2
 stdout: Logical volume "osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0
Running command: /usr/sbin/cryptsetup --key-file - luksOpen /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0 BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /bin/chown -R ceph:ceph /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
Running command: /bin/ln -s /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 /var/lib/ceph/osd/ceph-0/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQCWbD9bKrs6JxAA9WGlfT56+6EqogRIL1tDAQ==
 stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQCWbD9bKrs6JxAA9WGlfT56+6EqogRIL1tDAQ== with 0 caps)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid ea618678-b6d7-489a-9180-a7b67c3b69a0 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: /dev/sda

See the list:

[root@ceph-osd0 /]# ceph-volume lvm list /dev/sda

====== osd.0 =======

  [block]    /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0

      type                      block
      osd id                    0
      cluster fsid              0a9374ec-2998-47df-9a7f-d030e5b9c261
      cluster name              ceph
      osd fsid                  ea618678-b6d7-489a-9180-a7b67c3b69a0
      encrypted                 1
      cephx lockbox secret      AQCbbD9btcZ7NxAA1X+fydCuBVkyZZ6yZfymUw==
      block uuid                BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
      block device              /dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0
      vdo                       0
      crush device class        None
      devices                   /dev/sda

First attempt to zap:

[root@ceph-osd0 /]# umount /var/lib/ceph/osd/ceph-0

[root@ceph-osd0 /]# ceph-volume lvm zap /dev/sda
--> Zapping: /dev/sda
--> Closing encrypted path /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
Running command: /usr/sbin/cryptsetup remove /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
Running command: /usr/sbin/wipefs --all /dev/sda
 stderr: wipefs: error: /dev/sda: probing initialization failed: Device or resource busy
-->  RuntimeError: command returned non-zero exit status: 1

[root@ceph-osd0 /]# ps axuf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ceph      1616  0.1  3.8 845844 19276 ?        Ssl  12:31   0:05 ceph-osd -i 1 --setuser ceph --setgroup disk
root      1229  0.0  0.3  11820  1692 pts/4    Ss   12:04   0:00 bash
root      3328  0.0  0.3  51704  1696 pts/4    R+   13:25   0:00  \_ ps axuf
root       836  0.0  0.3  11820  1544 pts/3    Ss+  08:53   0:00 bash
root         5  0.0  0.1  11820   556 pts/1    Ss+  Jul05   0:00 bash
root         1  0.0  0.0   4360    24 ?        Ss   Jul05   0:00 sleep 365d

As we can see OSD.0 was not running, so there might be a timing issue in the removal sequence.

Second run fails too:

[root@ceph-osd0 /]# ceph-volume lvm zap /dev/sda
--> Zapping: /dev/sda
--> Closing encrypted path /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
Running command: /usr/sbin/cryptsetup remove /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
 stderr: Device /dev/mapper/BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16 is not active.
-->  RuntimeError: command returned non-zero exit status: 4

This means it's not able to resume the operation.

I wonder if this doesn't share so commonality with http://tracker.ceph.com/issues/24504

History

#1 Updated by Alfredo Deza over 5 years ago

  • Status changed from New to 4

Now that #24504 is completed, can you check if this is still an issue for you?

#2 Updated by Alfredo Deza over 5 years ago

Ping?

#3 Updated by Sébastien Han over 5 years ago

Sorry I don't have much time to test this again, I'll let you know once I resume my work.
Thanks for the patches.

#4 Updated by Alfredo Deza over 5 years ago

  • Status changed from 4 to Resolved

Also available in: Atom PDF