Actions
Bug #24952
closedceph-volume fails to list osds (KeyError: 'devices')
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
[2018-07-16 22:04:05,983][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:05,983][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"j6b8IV-KEw3-7bTw-Oqy8-1Ud3-juFC-SJBg12
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"IAjIC6-ScnM-tvH6-7BTy-TN31-hd82-bgDSzd
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0";"/dev/journals/journal1";"journal1";"journals";"obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=2b09dbb7-b12b-402f-9971-04855f0dd1a6,ceph.osd_fsid=41ba1d2c-2a5a-47b3-a0f9-fc7e6e274348,ceph.osd_id=0,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv1";"data-lv1";"test_group";"2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:06,394][ceph_volume.process][INFO ] Running command: /usr/sbin/blkid -t PARTUUID="2b09dbb7-b12b-402f-9971-04855f0dd1a6" -o device
[2018-07-16 22:04:06,829][ceph_volume.process][INFO ] stdout /dev/sdc1
[2018-07-16 22:04:06,829][ceph_volume.process][INFO ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-16 22:04:07,215][ceph_volume.process][INFO ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"j6b8IV-KEw3-7bTw-Oqy8-1Ud3-juFC-SJBg12
[2018-07-16 22:04:07,216][ceph_volume.process][INFO ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"IAjIC6-ScnM-tvH6-7BTy-TN31-hd82-bgDSzd
[2018-07-16 22:04:07,216][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0";"/dev/journals/journal1";"journal1";"journals";"obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y
[2018-07-16 22:04:07,216][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=2b09dbb7-b12b-402f-9971-04855f0dd1a6,ceph.osd_fsid=41ba1d2c-2a5a-47b3-a0f9-fc7e6e274348,ceph.osd_id=0,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv1";"data-lv1";"test_group";"2svevT-iOWB-NAlz-MjH5-X8zk-jbYj-Wl67HD
[2018-07-16 22:04:07,216][ceph_volume.process][INFO ] stdout ceph.cephx_lockbox_secret=,ceph.cluster_fsid=58fd7180-2887-4f01-9320-d194482712a2,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=obc6SR-cvkg-MbWh-cqC5-augU-hs4r-ha715Y,ceph.osd_fsid=106b105a-a7a2-49aa-901e-b2c8442f1c1a,ceph.osd_id=1,ceph.type=data,ceph.vdo=0";"/dev/test_group/data-lv2";"data-lv2";"test_group";"8E8pnM-AsEd-1lnI-P3DW-iLz9-PEt5-eChI3b
[2018-07-16 22:04:07,217][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 300, in main
self.list(args)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 120, in list
pretty_report(report)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/listing.py", line 53, in pretty_report
device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))
KeyError: 'devices'
Updated by Alfredo Deza almost 6 years ago
Can you expand on what circumstances you are producing this error?
Updated by Alfredo Deza almost 6 years ago
On the container side of things, it seems everything is using `--format=json` but this traceback is for pretty reporting. Is it happening for both?
Updated by Sébastien Han almost 6 years ago
This is happening when running "ceph-volume lvm list".
More info about the layout/setup:
[root@osd0 /]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 50G 0 disk sdb 8:16 0 50G 0 disk |-test_group-data--lv1 253:2 0 25G 0 lvm `-test_group-data--lv2 253:3 0 12.5G 0 lvm sdc 8:32 0 50G 0 disk |-sdc1 8:33 0 25G 0 part `-sdc2 8:34 0 25G 0 part `-journals-journal1 253:4 0 25G 0 lvm vda 252:0 0 41G 0 disk |-vda1 252:1 0 1M 0 part |-vda2 252:2 0 1G 0 part `-vda3 252:3 0 39G 0 part |-VolGroup00-LogVol00 253:0 0 37.5G 0 lvm /var/lib/ceph `-VolGroup00-LogVol01 253:1 0 1.5G 0 lvm [SWAP] [root@osd0 /]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert LogVol00 VolGroup00 -wi-ao---- <37.47g LogVol01 VolGroup00 -wi-ao---- 1.50g journal1 journals -wi-a----- <25.00g data-lv1 test_group -wi-a----- <25.00g data-lv2 test_group -wi-a----- 12.50g [root@osd0 /]# vgs VG #PV #LV #SN Attr VSize VFree VolGroup00 1 2 0 wz--n- <38.97g 0 journals 1 1 0 wz--n- <25.00g 0 test_group 1 2 0 wz--n- <50.00g 12.50g [root@osd0 /]# pvs PV VG Fmt Attr PSize PFree /dev/sdb test_group lvm2 a-- <50.00g 12.50g /dev/sdc2 journals lvm2 a-- <25.00g 0 /dev/vda3 VolGroup00 lvm2 a-- <38.97g 0
Updated by Sébastien Han almost 6 years ago
Note that when using --format=json the output works:
[root@osd0 /]# ceph-volume lvm list --format=json
{
"0": [
{
"devices": [
"/dev/sdb"
],
"lv_name": "data-lv1",
"lv_path": "/dev/test_group/data-lv1",
"lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv1,ceph.data_uuid=19Ld1t-6YO7-lj3O-WTEy-f
SCi-gPAC-BWQo6m,ceph.encrypted=0,ceph.journal_device=/dev/sdc1,ceph.journal_uuid=b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c,ceph.osd_fsid=1216d5e8-ac1f-45d1-af2a-eff43d375687,ceph.osd_id=0,ceph.type=data,ceph.vdo=0",
"lv_uuid": "19Ld1t-6YO7-lj3O-WTEy-fSCi-gPAC-BWQo6m",
"name": "data-lv1",
"path": "/dev/test_group/data-lv1",
"tags": {
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.data_device": "/dev/test_group/data-lv1",
"ceph.data_uuid": "19Ld1t-6YO7-lj3O-WTEy-fSCi-gPAC-BWQo6m",
"ceph.encrypted": "0",
"ceph.journal_device": "/dev/sdc1",
"ceph.journal_uuid": "b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c",
"ceph.osd_fsid": "1216d5e8-ac1f-45d1-af2a-eff43d375687",
"ceph.osd_id": "0",
"ceph.type": "data",
"ceph.vdo": "0"
},
"type": "data",
"vg_name": "test_group"
},
{
"path": "/dev/sdc1",
"tags": {
"PARTUUID": "b8df36ef-1a49-41a3-ad1d-9e2d7cf3f87c"
},
"type": "journal"
}
],
"1": [ [9/1613]
{
"devices": [
"/dev/sdc2"
],
"lv_name": "journal1",
"lv_path": "/dev/journals/journal1",
"lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=c0VdnC-dU5I-sPfV-UbC0-o
cjh-VG7Z-FjTI1Q,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO,ceph.osd_fsid=ed87126c-718d-4ab1-9350-4c1fc4b46b96,ceph.osd_id=1,ceph.type=journal,ceph.vdo=0",
"lv_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
"name": "journal1",
"path": "/dev/journals/journal1",
"tags": {
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.data_device": "/dev/test_group/data-lv2",
"ceph.data_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
"ceph.encrypted": "0",
"ceph.journal_device": "/dev/journals/journal1",
"ceph.journal_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
"ceph.osd_fsid": "ed87126c-718d-4ab1-9350-4c1fc4b46b96",
"ceph.osd_id": "1",
"ceph.type": "journal",
"ceph.vdo": "0"
},
"type": "journal",
"vg_name": "journals"
},
{
"devices": [
"/dev/sdb"
],
"lv_name": "data-lv2",
"lv_path": "/dev/test_group/data-lv2",
"lv_tags": "ceph.cephx_lockbox_secret=,ceph.cluster_fsid=05d71da5-97fc-47ea-baa4-8b3f2cfabc78,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.data_device=/dev/test_group/data-lv2,ceph.data_uuid=c0VdnC-dU5I-sPfV-UbC0-o
cjh-VG7Z-FjTI1Q,ceph.encrypted=0,ceph.journal_device=/dev/journals/journal1,ceph.journal_uuid=yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO,ceph.osd_fsid=ed87126c-718d-4ab1-9350-4c1fc4b46b96,ceph.osd_id=1,ceph.type=data,ceph.vdo=0",
"lv_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
"name": "data-lv2",
"path": "/dev/test_group/data-lv2",
"tags": {
"ceph.cephx_lockbox_secret": "",
"ceph.cluster_fsid": "05d71da5-97fc-47ea-baa4-8b3f2cfabc78",
"ceph.cluster_name": "ceph",
"ceph.crush_device_class": "None",
"ceph.data_device": "/dev/test_group/data-lv2",
"ceph.data_uuid": "c0VdnC-dU5I-sPfV-UbC0-ocjh-VG7Z-FjTI1Q",
"ceph.encrypted": "0",
"ceph.journal_device": "/dev/journals/journal1",
"ceph.journal_uuid": "yitj0u-VefX-fYxR-nAYa-Ddmy-QGLs-S6L7CO",
"ceph.osd_fsid": "ed87126c-718d-4ab1-9350-4c1fc4b46b96",
"ceph.osd_id": "1",
"ceph.type": "data",
"ceph.vdo": "0" },
"type": "data",
"vg_name": "test_group"
}
]
}
Updated by Alfredo Deza almost 6 years ago
- Status changed from New to 12
- Assignee set to Alfredo Deza
- Priority changed from Normal to Immediate
The problem here is that GPT partitions don't have devices, and the code assumes 'devices' is always going to be there. The fix is:
diff --git a/src/ceph-volume/ceph_volume/devices/lvm/listing.py b/src/ceph-volume/ceph_volume/devices/lvm/listing.py
index 6bc84a1686..40ea0c847b 100644
--- a/src/ceph-volume/ceph_volume/devices/lvm/listing.py
+++ b/src/ceph-volume/ceph_volume/devices/lvm/listing.py
@@ -49,8 +49,11 @@ def pretty_report(report):
value=value
)
)
- output.append(
- device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))
+ if not device.get('devices'):
+ continue
+ else:
+ output.append(
+ device_metadata_item_template.format(tag_name='devices', value=','.join(device['devices'])))
print(''.join(output))
Updated by Alfredo Deza almost 6 years ago
- Status changed from 12 to Resolved
Master PR https://github.com/ceph/ceph/pull/23129
Actions