Project

General

Profile

Bug #24952

ceph-volume fails to list osds (KeyError: 'devices')

Added by Sébastien Han about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Target version:
-
Start date:
07/16/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

[2018-07-16 22:04:05,983][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:05,983][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"j6b8IV-KEw3-7bTw-Oqy8-1Ud3-juFC-SJBg12
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"IAjIC6-ScnM-tvH6-7BTy-TN31-hd82-bgDSzd
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0";"/dev/journals/journal1";"journal1";"journals";"obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=2b09dbb7-b12b-402f-9971-04855f0dd1a6,ceph.osd_fsid=41ba1d2c-2a5a-47b3-a0f9-fc7e6e274348,ceph.osd_id=0,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv1";"data-lv1";"test_group";"2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:06,394][ceph_volume.process][INFO  ] Running command: /usr/sbin/blkid -t PARTUUID="2b09dbb7-b12b-402f-9971-04855f0dd1a6" -o device
[2018-07-16 22:04:06,829][ceph_volume.process][INFO  ] stdout /dev/sdc1
[2018-07-16 22:04:06,829][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-16 22:04:07,215][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"j6b8IV-KEw3-7bTw-Oqy8-1Ud3-juFC-SJBg12
[2018-07-16 22:04:07,216][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"IAjIC6-ScnM-tvH6-7BTy-TN31-hd82-bgDSzd
[2018-07-16 22:04:07,216][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0";"/dev/journals/journal1";"journal1";"journals";"obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y
[2018-07-16 22:04:07,216][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=2b09dbb7-b12b-402f-9971-04855f0dd1a6,ceph.osd_fsid=41ba1d2c-2a5a-47b3-a0f9-fc7e6e274348,ceph.osd_id=0,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv1";"data-lv1";"test_group";"2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD
[2018-07-16 22:04:07,216][ceph_volume.process][INFO  ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:07,217][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 300, in main
    self.list(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 120, in list
    pretty_report(report)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 53, in pretty_report
    device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))
KeyError: 'devices'

History

#1 Updated by Alfredo Deza about 1 year ago

Can you expand on what circumstances you are producing this error?

#2 Updated by Alfredo Deza about 1 year ago

On the container side of things, it seems everything is using `--format=json` but this traceback is for pretty reporting. Is it happening for both?

#3 Updated by Sébastien Han about 1 year ago

This is happening when running "ceph-volume lvm list".

More info about the layout/setup:

[root@osd0 /]# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    0   50G  0 disk
sdb                       8:16   0   50G  0 disk
|-test_group-data--lv1  253:2    0   25G  0 lvm
`-test_group-data--lv2  253:3    0 12.5G  0 lvm
sdc                       8:32   0   50G  0 disk
|-sdc1                    8:33   0   25G  0 part
`-sdc2                    8:34   0   25G  0 part
  `-journals-journal1   253:4    0   25G  0 lvm
vda                     252:0    0   41G  0 disk
|-vda1                  252:1    0    1M  0 part
|-vda2                  252:2    0    1G  0 part
`-vda3                  252:3    0   39G  0 part
  |-VolGroup00-LogVol00 253:0    0 37.5G  0 lvm  /var/lib/ceph
  `-VolGroup00-LogVol01 253:1    0  1.5G  0 lvm  [SWAP]
[root@osd0 /]# lvs
  LV       VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  LogVol00 VolGroup00 -wi-ao---- <37.47g
  LogVol01 VolGroup00 -wi-ao----   1.50g
  journal1 journals   -wi-a----- <25.00g
  data-lv1 test_group -wi-a----- <25.00g
  data-lv2 test_group -wi-a-----  12.50g
[root@osd0 /]# vgs
  VG         #PV #LV #SN Attr   VSize   VFree
  VolGroup00   1   2   0 wz--n- <38.97g     0
  journals     1   1   0 wz--n- <25.00g     0
  test_group   1   2   0 wz--n- <50.00g 12.50g
[root@osd0 /]# pvs
  PV         VG         Fmt  Attr PSize   PFree
  /dev/sdb   test_group lvm2 a--  <50.00g 12.50g
  /dev/sdc2  journals   lvm2 a--  <25.00g     0
  /dev/vda3  VolGroup00 lvm2 a--  <38.97g     0

#4 Updated by Sébastien Han about 1 year ago

Note that when using --format=json the output works:

[root@osd0 /]# ceph-volume lvm list --format=json
{
    "0": [
        {
            "devices": [
                "/dev/sdb" 
            ],
            "lv_name": "data-lv1",
            "lv_path": "/dev/test_group/data-lv1",
            "lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=19Ld1t-6YO7-lj3O-WTEy-f
SCi-gPAC-BWQo6m,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c,ceph.osd_fsid=1216d5e8-ac1f-45d1-af2a-eff43d375687,ceph.osd_id=0,ceph.type=data,ceph.vdo=0",
            "lv_uuid": "19Ld1t-6YO7-lj3O-WTEy-fSCi-gPAC-BWQo6m",
            "name": "data-lv1",
            "path": "/dev/test_group/data-lv1",
            "tags": {
                "ceph.cephx_lockbox_secret": "",
                "ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
                "ceph.cluster_name": "ceph",
                "ceph.crush_device_class": "None",
                "ceph.data_device": "/dev/test_group/data-lv1",
                "ceph.data_uuid": "19Ld1t-6YO7-lj3O-WTEy-fSCi-gPAC-BWQo6m",
                "ceph.encrypted": "0",
                "ceph.journal_device": "/dev/sdc1",
                "ceph.journal_uuid": "b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c",
                "ceph.osd_fsid": "1216d5e8-ac1f-45d1-af2a-eff43d375687",
                "ceph.osd_id": "0",
                "ceph.type": "data",
                "ceph.vdo": "0" 
            },
            "type": "data",
            "vg_name": "test_group" 
        },
        {
            "path": "/dev/sdc1",
            "tags": {
                "PARTUUID": "b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c" 
            },
            "type": "journal" 
        }
    ],
"1": [                                                                                                                                                                                                                            [9/1613]
        {
            "devices": [
                "/dev/sdc2" 
            ],
            "lv_name": "journal1",
            "lv_path": "/dev/journals/journal1",
            "lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=c0VdnC-dU5I-sPfV-UbC0-o
cjh-VG7Z-FjTI1Q,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO,ceph.osd_fsid=ed87126c-718d-4ab1-9350-4c1fc4b46b96,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0",
            "lv_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
            "name": "journal1",
            "path": "/dev/journals/journal1",
            "tags": {
                "ceph.cephx_lockbox_secret": "",
                "ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
                "ceph.cluster_name": "ceph",
                "ceph.crush_device_class": "None",
                "ceph.data_device": "/dev/test_group/data-lv2",
                "ceph.data_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
                "ceph.encrypted": "0",
                "ceph.journal_device": "/dev/journals/journal1",
                "ceph.journal_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
                "ceph.osd_fsid": "ed87126c-718d-4ab1-9350-4c1fc4b46b96",
                "ceph.osd_id": "1",
                "ceph.type": "journal",
                "ceph.vdo": "0" 
            },
            "type": "journal",
            "vg_name": "journals" 
        },
        {
            "devices": [
                "/dev/sdb" 
            ],
            "lv_name": "data-lv2",
            "lv_path": "/dev/test_group/data-lv2",
            "lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=c0VdnC-dU5I-sPfV-UbC0-o
cjh-VG7Z-FjTI1Q,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO,ceph.osd_fsid=ed87126c-718d-4ab1-9350-4c1fc4b46b96,ceph.osd_id=1,ceph.type=data,ceph.vdo=0",
            "lv_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
            "name": "data-lv2",
            "path": "/dev/test_group/data-lv2",
            "tags": {
                "ceph.cephx_lockbox_secret": "",
                "ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
                "ceph.cluster_name": "ceph",
                "ceph.crush_device_class": "None",
                "ceph.data_device": "/dev/test_group/data-lv2",
                "ceph.data_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
                "ceph.encrypted": "0",
                "ceph.journal_device": "/dev/journals/journal1",
                "ceph.journal_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
                "ceph.osd_fsid": "ed87126c-718d-4ab1-9350-4c1fc4b46b96",
                "ceph.osd_id": "1",
                "ceph.type": "data",
                "ceph.vdo": "0"            },
            "type": "data",
            "vg_name": "test_group" 
        }
    ]
}

#5 Updated by Alfredo Deza about 1 year ago

  • Status changed from New to Verified
  • Assignee set to Alfredo Deza
  • Priority changed from Normal to Immediate

The problem here is that GPT partitions don't have devices, and the code assumes 'devices' is always going to be there. The fix is:

diff --git a/src/ceph-volume/ceph_volume/devices/lvm/listing.py b/src/ceph-volume/ceph_volume/devices/lvm/listing.py
index 6bc84a1686..40ea0c847b 100644
--- a/src/ceph-volume/ceph_volume/devices/lvm/listing.py
+++ b/src/ceph-volume/ceph_volume/devices/lvm/listing.py
@@ -49,8 +49,11 @@ def pretty_report(report):
                         value=value
                     )
                 )
-            output.append(
-                device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))
+            if not device.get('devices'):
+                continue
+            else:
+                output.append(
+                    device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))

     print(''.join(output))

#6 Updated by Alfredo Deza about 1 year ago

  • Status changed from Verified to Resolved

Also available in: Atom PDF