Bug #24784
closedMultiplePVs errors when trying to zap
0%
Description
Was trying to deploy, zap, redeploy again, and was getting MultiplePVs errors on one device. Stepping through
it seems that PVolumes() can have multiple devices repeated:
ipdb> api.get_pv(pv_name=device) *** MultiplePVsError: Got more than 1 result looking for physical volume: /dev/sdh ipdb> device '/dev/sdh' ipdb> pvs = api.PVolumes() ipdb> pvs [</dev/nvme0n1>, </dev/nvme0n2>, </dev/sda5>, </dev/sda5>, </dev/sda5>, </dev/sdb>, </dev/sdd>, </dev/sdd>, </dev/sde>, </dev/sdf>, </dev/sdf>, </dev/sdg>, </dev/sdh>, </dev/sdh>, </dev/sdi>] ipdb> pvs[-3] </dev/sdh> ipdb> pvs[-2] </dev/sdh> ipdb> pvs[-2].pv_api {'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': ''} ipdb> pvs[-3].pv_api {'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': 'DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4'} ipdb> for pv in pvs: print pv.pv_api {'pv_uuid': 'n3Y1Nv-rAb3-0oHR-ggzY-aW1G-ewkV-ccScKh', 'pv_tags': '', 'pv_name': '/dev/nvme0n1', 'vg_name': '', 'lv_uuid': ''} {'pv_uuid': 'Rn5Xcj-fckQ-iWsC-napy-Fif1-u2b6-ODhCJE', 'pv_tags': '', 'pv_name': '/dev/nvme0n2', 'vg_name': '', 'lv_uuid': ''} {'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': 'Z2gyTg-bQau-GvZb-Uqty-WHur-xcP3-v2usYb'} {'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': 'fbzgJ6-BqEm-Rb99-ou3N-gqek-liGR-p2ZI6h'} {'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': ''} {'pv_uuid': 'fAkk0U-iNUT-caYW-dUD8-rZdz-Jgeb-x7mfp2', 'pv_tags': '', 'pv_name': '/dev/sdb', 'vg_name': 'ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48', 'lv_uuid': 'wee3Mq-CSGR-Bql4-RoRg-PUog-429A-VMprL3'} {'pv_uuid': 'oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU', 'pv_tags': '', 'pv_name': '/dev/sdd', 'vg_name': 'ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea', 'lv_uuid': 'KS6gWC-PLcC-IZ1s-0hiG-qxnw-x4cY-GYSx0s'} {'pv_uuid': 'oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU', 'pv_tags': '', 'pv_name': '/dev/sdd', 'vg_name': 'ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea', 'lv_uuid': ''} {'pv_uuid': 'zcRtZL-lRlQ-jLOr-x8GV-X18N-0At2-C5JM1c', 'pv_tags': '', 'pv_name': '/dev/sde', 'vg_name': 'ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603', 'lv_uuid': ''} {'pv_uuid': 'JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8', 'pv_tags': '', 'pv_name': '/dev/sdf', 'vg_name': 'ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a', 'lv_uuid': 'LJRpGc-r79Q-H93X-CvaO-IfDk-njrT-29PZo6'} {'pv_uuid': 'JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8', 'pv_tags': '', 'pv_name': '/dev/sdf', 'vg_name': 'ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a', 'lv_uuid': ''} {'pv_uuid': 'B8XcrG-6Y42-bwC5-Um4w-ELz5-1q4d-T5s4Sx', 'pv_tags': '', 'pv_name': '/dev/sdg', 'vg_name': '', 'lv_uuid': ''} {'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': 'DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4'} {'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': ''} {'pv_uuid': 'VAJcIE-3hWT-B6aS-TAqx-5Tpl-Pqi3-3ZfFgM', 'pv_tags': '', 'pv_name': '/dev/sdi', 'vg_name': 'ceph-1dc6a712-3aad-4324-acf1-c369b58a6550', 'lv_uuid': ''}
Calling out to `pvs` confirms this behavior coming from LVM:
root@node9:/home/vagrant# pvs --no-heading --readonly -o pv_name,pv_tags,pv_uuid,vg_name,lv_uuid /dev/nvme0n1 n3Y1Nv-rAb3-0oHR-ggzY-aW1G-ewkV-ccScKh /dev/nvme0n2 Rn5Xcj-fckQ-iWsC-napy-Fif1-u2b6-ODhCJE /dev/sda5 n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg Z2gyTg-bQau-GvZb-Uqty-WHur-xcP3-v2usYb /dev/sda5 n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg fbzgJ6-BqEm-Rb99-ou3N-gqek-liGR-p2ZI6h /dev/sda5 n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg /dev/sdb fAkk0U-iNUT-caYW-dUD8-rZdz-Jgeb-x7mfp2 ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48 wee3Mq-CSGR-Bql4-RoRg-PUog-429A-VMprL3 /dev/sdd oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea KS6gWC-PLcC-IZ1s-0hiG-qxnw-x4cY-GYSx0s /dev/sdd oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea /dev/sde zcRtZL-lRlQ-jLOr-x8GV-X18N-0At2-C5JM1c ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603 /dev/sdf JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8 ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a LJRpGc-r79Q-H93X-CvaO-IfDk-njrT-29PZo6 /dev/sdf JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8 ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a /dev/sdg B8XcrG-6Y42-bwC5-Um4w-ELz5-1q4d-T5s4Sx /dev/sdh 1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4 /dev/sdh 1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b /dev/sdi VAJcIE-3hWT-B6aS-TAqx-5Tpl-Pqi3-3ZfFgM ceph-1dc6a712-3aad-4324-acf1-c369b58a6550
One can see that, if we remove the lv_* fields, the devices are back to unique again:
root@node9:/home/vagrant# pvs --no-heading --readonly /dev/nvme0n1 lvm2 --- 100.00g 100.00g /dev/nvme0n2 lvm2 --- 58.18g 58.18g /dev/sda5 ubuntubox-vg lvm2 a-- 299.52g 12.00m /dev/sdb ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48 lvm2 a-- 10.74g 0 /dev/sdd ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea lvm2 a-- 10.74g 756.00m /dev/sde ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603 lvm2 a-- 10.74g 10.74g /dev/sdf ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a lvm2 a-- 10.74g 756.00m /dev/sdg lvm2 --- 10.74g 10.74g /dev/sdh ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b lvm2 a-- 10.74g 756.00m /dev/sdi ceph-1dc6a712-3aad-4324-acf1-c369b58a6550 lvm2 a-- 10.74g 10.74g
This happens because when creating an LV that does not use 100%FREE, the LVM engine will report that the PV is available (open/free) for more LVs (e.g. if there is 10MB left to create another LV).
This issue is two-pronged:
1. The API should be smart enough to detect this case (PV has multiple entries for same VG) and avoid this issue.
2. The zap command should be more lenient (and maybe add some logic) when a PV is part of multiple VGs
Updated by Alfredo Deza almost 6 years ago
My half-baked solution allowed me to zap these pvs:
diff --git a/src/ceph-volume/ceph_volume/devices/lvm/zap.py b/src/ceph-volume/ceph_volume/devices/lvm/zap.py index ee5cdf97b6..4fc887fa87 100644 --- a/src/ceph-volume/ceph_volume/devices/lvm/zap.py +++ b/src/ceph-volume/ceph_volume/devices/lvm/zap.py @@ -62,7 +62,12 @@ class Zap(object): # check if there was a pv created with the # name of device - pv = api.get_pv(pv_name=device) + pvs = [pv for pv in api.PVolumes() if pv.name == device] + if pvs: + pv = pvs[0] + else: + pv = None + #pv = api.get_pv(pv_name=device) if pv: vg_name = pv.vg_name lv = api.get_lv(vg_name=vg_name)
It is a hacky patch, but should give an idea of what is it that we need here to make this work (reduce the multiple pvs into just one if it matches the name)