Project

General

Profile

Actions
Copy link

Bug #24784

closed

MultiplePVs errors when trying to zap

Added by Alfredo Deza almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Andrew Schoen
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Was trying to deploy, zap, redeploy again, and was getting MultiplePVs errors on one device. Stepping through
it seems that PVolumes() can have multiple devices repeated:

ipdb> api.get_pv(pv_name=device)
*** MultiplePVsError: Got more than 1 result looking for physical volume: /dev/sdh
ipdb> device
'/dev/sdh'
ipdb> pvs = api.PVolumes()
ipdb> pvs
[</dev/nvme0n1>, </dev/nvme0n2>, </dev/sda5>, </dev/sda5>, </dev/sda5>, </dev/sdb>, </dev/sdd>, </dev/sdd>, </dev/sde>, </dev/sdf>, </dev/sdf>, </dev/sdg>, </dev/sdh>, </dev/sdh>, </dev/sdi>]
ipdb> pvs[-3]
</dev/sdh>
ipdb> pvs[-2]
</dev/sdh>
ipdb> pvs[-2].pv_api
{'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': ''}
ipdb> pvs[-3].pv_api
{'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': 'DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4'}
ipdb> for pv in pvs: print pv.pv_api
{'pv_uuid': 'n3Y1Nv-rAb3-0oHR-ggzY-aW1G-ewkV-ccScKh', 'pv_tags': '', 'pv_name': '/dev/nvme0n1', 'vg_name': '', 'lv_uuid': ''}
{'pv_uuid': 'Rn5Xcj-fckQ-iWsC-napy-Fif1-u2b6-ODhCJE', 'pv_tags': '', 'pv_name': '/dev/nvme0n2', 'vg_name': '', 'lv_uuid': ''}
{'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': 'Z2gyTg-bQau-GvZb-Uqty-WHur-xcP3-v2usYb'}
{'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': 'fbzgJ6-BqEm-Rb99-ou3N-gqek-liGR-p2ZI6h'}
{'pv_uuid': 'n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv', 'pv_tags': '', 'pv_name': '/dev/sda5', 'vg_name': 'ubuntubox-vg', 'lv_uuid': ''}
{'pv_uuid': 'fAkk0U-iNUT-caYW-dUD8-rZdz-Jgeb-x7mfp2', 'pv_tags': '', 'pv_name': '/dev/sdb', 'vg_name': 'ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48', 'lv_uuid': 'wee3Mq-CSGR-Bql4-RoRg-PUog-429A-VMprL3'}
{'pv_uuid': 'oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU', 'pv_tags': '', 'pv_name': '/dev/sdd', 'vg_name': 'ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea', 'lv_uuid': 'KS6gWC-PLcC-IZ1s-0hiG-qxnw-x4cY-GYSx0s'}
{'pv_uuid': 'oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU', 'pv_tags': '', 'pv_name': '/dev/sdd', 'vg_name': 'ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea', 'lv_uuid': ''}
{'pv_uuid': 'zcRtZL-lRlQ-jLOr-x8GV-X18N-0At2-C5JM1c', 'pv_tags': '', 'pv_name': '/dev/sde', 'vg_name': 'ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603', 'lv_uuid': ''}
{'pv_uuid': 'JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8', 'pv_tags': '', 'pv_name': '/dev/sdf', 'vg_name': 'ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a', 'lv_uuid': 'LJRpGc-r79Q-H93X-CvaO-IfDk-njrT-29PZo6'}
{'pv_uuid': 'JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8', 'pv_tags': '', 'pv_name': '/dev/sdf', 'vg_name': 'ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a', 'lv_uuid': ''}
{'pv_uuid': 'B8XcrG-6Y42-bwC5-Um4w-ELz5-1q4d-T5s4Sx', 'pv_tags': '', 'pv_name': '/dev/sdg', 'vg_name': '', 'lv_uuid': ''}
{'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': 'DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4'}
{'pv_uuid': '1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo', 'pv_tags': '', 'pv_name': '/dev/sdh', 'vg_name': 'ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b', 'lv_uuid': ''}
{'pv_uuid': 'VAJcIE-3hWT-B6aS-TAqx-5Tpl-Pqi3-3ZfFgM', 'pv_tags': '', 'pv_name': '/dev/sdi', 'vg_name': 'ceph-1dc6a712-3aad-4324-acf1-c369b58a6550', 'lv_uuid': ''}

Calling out to `pvs` confirms this behavior coming from LVM:

root@node9:/home/vagrant# pvs --no-heading --readonly -o pv_name,pv_tags,pv_uuid,vg_name,lv_uuid
  /dev/nvme0n1         n3Y1Nv-rAb3-0oHR-ggzY-aW1G-ewkV-ccScKh
  /dev/nvme0n2         Rn5Xcj-fckQ-iWsC-napy-Fif1-u2b6-ODhCJE
  /dev/sda5            n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg                              Z2gyTg-bQau-GvZb-Uqty-WHur-xcP3-v2usYb
  /dev/sda5            n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg                              fbzgJ6-BqEm-Rb99-ou3N-gqek-liGR-p2ZI6h
  /dev/sda5            n9nk2f-WzLi-OMxx-rg3I-N1S8-A6sY-gEMZmv ubuntubox-vg
  /dev/sdb             fAkk0U-iNUT-caYW-dUD8-rZdz-Jgeb-x7mfp2 ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48 wee3Mq-CSGR-Bql4-RoRg-PUog-429A-VMprL3
  /dev/sdd             oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea KS6gWC-PLcC-IZ1s-0hiG-qxnw-x4cY-GYSx0s
  /dev/sdd             oDArKA-c7Uo-DrZr-G811-NgtY-RULO-RQ9ElU ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea
  /dev/sde             zcRtZL-lRlQ-jLOr-x8GV-X18N-0At2-C5JM1c ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603
  /dev/sdf             JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8 ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a LJRpGc-r79Q-H93X-CvaO-IfDk-njrT-29PZo6
  /dev/sdf             JXKY0R-POQ7-NVmL-eYQX-BwsN-B6UV-fdW5u8 ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a
  /dev/sdg             B8XcrG-6Y42-bwC5-Um4w-ELz5-1q4d-T5s4Sx
  /dev/sdh             1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b DdQ6dE-SNuE-XDHM-YEaw-S6Ow-Ttr4-ixxud4
  /dev/sdh             1maJNA-3hEk-9urD-gwgU-pBXo-xiKj-FoeZYo ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b
  /dev/sdi             VAJcIE-3hWT-B6aS-TAqx-5Tpl-Pqi3-3ZfFgM ceph-1dc6a712-3aad-4324-acf1-c369b58a6550

One can see that, if we remove the lv_* fields, the devices are back to unique again:

root@node9:/home/vagrant# pvs --no-heading --readonly
  /dev/nvme0n1                                           lvm2 ---  100.00g 100.00g
  /dev/nvme0n2                                           lvm2 ---   58.18g  58.18g
  /dev/sda5    ubuntubox-vg                              lvm2 a--  299.52g  12.00m
  /dev/sdb     ceph-e1a09bf1-7ea2-4c6a-b127-4b1469742e48 lvm2 a--   10.74g      0
  /dev/sdd     ceph-a08bf0e9-6fe8-44e4-9b26-743c9fbc92ea lvm2 a--   10.74g 756.00m
  /dev/sde     ceph-2b99f2b8-8f61-48cd-9cf2-ea1f6654a603 lvm2 a--   10.74g  10.74g
  /dev/sdf     ceph-e0eeceaa-180d-48e0-b189-93d32aa31a6a lvm2 a--   10.74g 756.00m
  /dev/sdg                                               lvm2 ---   10.74g  10.74g
  /dev/sdh     ceph-5af96244-87c7-4eb2-a3fc-5c3b5241c17b lvm2 a--   10.74g 756.00m
  /dev/sdi     ceph-1dc6a712-3aad-4324-acf1-c369b58a6550 lvm2 a--   10.74g  10.74g

This happens because when creating an LV that does not use 100%FREE, the LVM engine will report that the PV is available (open/free) for more LVs (e.g. if there is 10MB left to create another LV).

This issue is two-pronged:

1. The API should be smart enough to detect this case (PV has multiple entries for same VG) and avoid this issue.
2. The zap command should be more lenient (and maybe add some logic) when a PV is part of multiple VGs

Actions
Copy link
 #1

Updated by Alfredo Deza almost 6 years ago

My half-baked solution allowed me to zap these pvs:

diff --git a/src/ceph-volume/ceph_volume/devices/lvm/zap.py b/src/ceph-volume/ceph_volume/devices/lvm/zap.py
index ee5cdf97b6..4fc887fa87 100644
--- a/src/ceph-volume/ceph_volume/devices/lvm/zap.py
+++ b/src/ceph-volume/ceph_volume/devices/lvm/zap.py
@@ -62,7 +62,12 @@ class Zap(object):

         # check if there was a pv created with the
         # name of device
-        pv = api.get_pv(pv_name=device)
+        pvs = [pv for pv in api.PVolumes() if pv.name == device]
+        if pvs:
+            pv = pvs[0]
+        else:
+            pv = None
+        #pv = api.get_pv(pv_name=device)
         if pv:
             vg_name = pv.vg_name
             lv = api.get_lv(vg_name=vg_name)

It is a hacky patch, but should give an idea of what is it that we need here to make this work (reduce the multiple pvs into just one if it matches the name)

Actions
Copy link
 #3

Updated by Andrew Schoen over 5 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF