Actions
Bug #48445
openceph-volume lvm zap fails to properly dismantle volume groups
% Done:
0%
Source:
Community (user)
Tags:
ceph-volume,cephadm
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
After successfully deploying with cephadm, and then successfully purging the cluster with cephadm, I tried to use zap on the OSD drives to reuse them again for another deployment.
Zap was able to identify the drives and successfully run dd, but removal of the vg failed:
$> ceph-volume lvm zap --force --osd-id 0 --> Zapping: /dev/sdb --> Zapping lvm member /dev/sdb. lv_path is /dev/ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b/osd-data-aa18cf0f-31dc-497b-819f-177fe4aeab86 Running command: /bin/dd if=/dev/zero of=/dev/ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b/osd-data-aa18cf0f-31dc-497b-819f-177fe4aeab86 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149273 s, 702 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b Running command: /sbin/vgremove -v -f ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b stderr: VG name on command line not found in list of VGs: ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b Volume group "ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b" not found Cannot process volume group ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b --> Unable to remove vg ceph-6763dfae-8212-40dc-8ef8-9e2b492caf9b --> RuntimeError: command returned non-zero exit status: 5
This behavior was the same for all OSD hosts. On the same host that returned the error message above, lsblk returns:
$> lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 10G 0 disk └─sda1 8:1 0 10G 0 part / sdb 8:16 0 20G 0 disk └─ceph--6763dfae--8212--40dc--8ef8--9e2b492caf9b-osd--data--aa18cf0f--31dc--497b--819f--177fe4aeab86 253:0 0 20G 0 lvm sdc 8:32 0 20G 0 disk └─ceph--43baa65a--90a7--43a0--b29a--a7837c7b97d0-osd--data--2f46c30d--4bca--42ea--9fc9--eab44aa092b6 253:1 0 20G 0 lvm sdd 8:48 0 20G 0 disk └─ceph--7a7e3a17--3783--41be--b744--cf04d8dc40c7-osd--data--7867f469--5c77--44a8--b4d2--cc45f79dbcb8 253:2 0 20G 0 lvm sde 8:64 0 20G 0 disk └─ceph--2963fdd0--e2c0--413f--a7df--62c97c17cbac-osd--data--269c29db--bd1d--4881--b3ff--6e62b05623e1 253:3 0 20G 0 lvm sdf 8:80 0 20G 0 disk └─ceph--ff5f06c8--bc7d--4728--b674--7575156ce344-osd--data--963656a8--cb35--4768--9535--4ad622719aa1 253:4 0 20G 0 lvm sdg 8:96 0 20G 0 disk └─ceph--eb7bff0b--76af--4120--b606--cb63dd061763-osd--data--10f1222a--ac0f--471d--afba--84ce8999b23e 253:5 0 20G 0 lvm sr0 11:0 1 951M 0 rom
The volume is still there, but lvm does not seem to be able to see anything:
$> lvm pvs $> lvm lvs $> lvm vgs
This would be a minor annoyance if it is just some artifact overlooked while purging, but it prevents the drive from being used again by the orchestrator. Deploying the exact same cluster congfiguration to the same machines, the orchestrator reports that there are stray daemons and locked/unavailable disks:
$> ceph orch device ls HOST PATH TYPE SIZE DEVICE_ID MODEL VENDOR ROTATIONAL AVAIL REJECT REASONS Mon0 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked OSD0 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked OSD0 /dev/sdb hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD0 /dev/sdc hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD0 /dev/sdd hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD0 /dev/sde hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD0 /dev/sdf hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD0 /dev/sdg hdd 20.0G Virtual disk VMware 1 False locked, Insufficient space (<5GB) on vgs, LVM detected OSD1 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked OSD1 /dev/sdb hdd 20.0G Virtual disk VMware 1 False locked OSD1 /dev/sdc hdd 20.0G Virtual disk VMware 1 False locked OSD1 /dev/sdd hdd 20.0G Virtual disk VMware 1 False locked OSD1 /dev/sde hdd 20.0G Virtual disk VMware 1 False locked OSD1 /dev/sdf hdd 20.0G Virtual disk VMware 1 False locked OSD1 /dev/sdg hdd 20.0G Virtual disk VMware 1 False locked OSD2 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked OSD2 /dev/sdb hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked OSD2 /dev/sdc hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked OSD2 /dev/sdd hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked OSD2 /dev/sde hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked OSD2 /dev/sdf hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked OSD2 /dev/sdg hdd 20.0G Virtual disk VMware 1 False Insufficient space (<5GB) on vgs, LVM detected, locked MDS0 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked MDS1 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked MDS2 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked Mon1 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked Mon2 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked Client0 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked Client1 /dev/sda hdd 10.0G Virtual disk VMware 1 False locked
No data to display
Actions