Project

General

Profile

Bug #24796

ceph-volume fails with KeyError: 'ceph.cluster_name'

Added by S├ębastien Han 12 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
07/06/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

I'm trying to purge /dev/sdb

[root@ceph-osd0 /]# ceph-volume lvm zap /dev/sdb --destroy
--> Zapping: /dev/sdb
-->  KeyError: 'ceph.cluster_name'

History

#1 Updated by Alfredo Deza 12 months ago

can you please add the relevant log output for this call (/var/log/ceph/ceph-volume.log) or rerun with:

CEPH_VOLUME_DEBUG=1 ceph-volume lvm zap /dev/sdb --destroy

And paste that output

#2 Updated by S├ębastien Han 12 months ago

[2018-07-06 14:36:21,056][ceph_volume.main][INFO  ] Running command: ceph-volume  lvm zap /dev/sdb
[2018-07-06 14:36:21,056][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-06 14:36:21,285][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"gqFct1-q2eW-5I7a-IQl2-g2Vm-Nsiq-B2CqHo
[2018-07-06 14:36:21,285][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"T8lHub-uZnx-CX9A-354Z-Jd3P-fTPq-dve374
[2018-07-06 14:36:21,285][ceph_volume.process][INFO  ] stdout ceph.block_device=/dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0,ceph.block_uuid=BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16,ceph.cephx_lockbox_secret=AQCbbD9btcZ7NxAA1X+fydCuBVkyZZ6yZfymUw==,ceph.cluster_fsid=0a9374ec-2998-47df-9a7f-d030e5b9c261,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=ea618678-b6d7-489a-9180-a7b67c3b69a0,ceph.osd_id=0,ceph.type=block,ceph.vdo=0";"/dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0";"osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0";"ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2";"BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
[2018-07-06 14:36:21,285][ceph_volume.process][INFO  ] stdout ceph.block_device=/dev/ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7/osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d,ceph.type=block";"/dev/ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7/osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d";"osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d";"ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7";"iQwkRD-8xsr-Gwzi-YcUe-z93X-48Vy-w923Gn[2018-07-06 14:36:21,285][ceph_volume.devices.lvm.zap][INFO  ] Zapping: /dev/sdb
[2018-07-06 14:36:21,286][ceph_volume.process][INFO  ] Running command: /usr/sbin/pvs --no-heading --readonly --separator=";" -o pv_name,pv_tags,pv_uuid,vg_name,lv_uuid
[2018-07-06 14:36:21,514][ceph_volume.process][INFO  ] stdout /dev/sda";"";"ijcyzY-sgjK-cCfq-aeoB-FstK-Z2Ri-tyzNBr";"ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2";"BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
[2018-07-06 14:36:21,514][ceph_volume.process][INFO  ] stdout /dev/sdb";"";"9zlKVn-J1A8-zwAV-XLKz-gpb6-Wave-IUpQuF";"ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7";"iQwkRD-8xsr-Gwzi-YcUe-z93X-48Vy-w923Gn[2018-07-06 14:36:21,514][ceph_volume.process][INFO  ] stdout /dev/vda3";"";"qGTgpA-Amb9-J2GQ-bP3A-2X6n-g5Ud-zPWx6a";"VolGroup00";"gqFct1-q2eW-5I7a-IQl2-g2Vm-Nsiq-B2CqHo
[2018-07-06 14:36:21,514][ceph_volume.process][INFO  ] stdout /dev/vda3";"";"qGTgpA-Amb9-J2GQ-bP3A-2X6n-g5Ud-zPWx6a";"VolGroup00";"T8lHub-uZnx-CX9A-354Z-Jd3P-fTPq-dve374[2018-07-06 14:36:21,514][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvs --noheadings --readonly --separator=";" -o lv_tags,lv_path,lv_name,vg_name,lv_uuid
[2018-07-06 14:36:21,741][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol00";"LogVol00";"VolGroup00";"gqFct1-q2eW-5I7a-IQl2-g2Vm-Nsiq-B2CqHo
[2018-07-06 14:36:21,741][ceph_volume.process][INFO  ] stdout ";"/dev/VolGroup00/LogVol01";"LogVol01";"VolGroup00";"T8lHub-uZnx-CX9A-354Z-Jd3P-fTPq-dve374
[2018-07-06 14:36:21,741][ceph_volume.process][INFO  ] stdout ceph.block_device=/dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0,ceph.block_uuid=BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16,ceph.cephx_lockbox_secret=AQCbbD9btcZ7NxAA1X+fydCuBVkyZZ6yZfymUw==,ceph.cluster_fsid=0a9374ec-2998-47df-9a7f-d030e5b9c261,ceph.cluster_name=ceph,ceph.crush_device_class=None,ceph.encrypted=1,ceph.osd_fsid=ea618678-b6d7-489a-9180-a7b67c3b69a0,ceph.osd_id=0,ceph.type=block,ceph.vdo=0";"/dev/ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2/osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0";"osd-block-ea618678-b6d7-489a-9180-a7b67c3b69a0";"ceph-4d5af038-df98-437a-b038-a2f3f45d0cd2";"BTOKvd-zxmP-58uL-CAy6-coEt-Ir8e-VDdV16
[2018-07-06 14:36:21,741][ceph_volume.process][INFO  ] stdout ceph.block_device=/dev/ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7/osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d,ceph.type=block";"/dev/ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7/osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d";"osd-block-5fafadcf-662a-47cf-9bb4-f1912fa8e33d";"ceph-fca417c9-7960-4627-82c9-5954d7d8e7b7";"iQwkRD-8xsr-Gwzi-YcUe-z93X-48Vy-w923Gn[2018-07-06 14:36:21,741][ceph_volume][ERROR ] exception caught by decoratorTraceback (most recent call last):  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/zap.py", line 169, in main
    self.zap(args)
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/zap.py", line 73, in zap
    osd_path = "/var/lib/ceph/osd/{}-{}".format(lv.tags['ceph.cluster_name'], lv.tags['ceph.osd_id'])
KeyError: 'ceph.cluster_name'

#3 Updated by Alfredo Deza 12 months ago

  • Status changed from New to Verified

Yeah, I can hit the same in some situations, thanks for reporting this!

Adding CLI output:

(tmp) root@node9:/home/vagrant# CEPH_VOLUME_DEBUG=1 ceph-volume lvm zap --destroy /dev/sdf
--> Zapping: /dev/sdf
Traceback (most recent call last):
  File "/home/vagrant/tmp/bin/ceph-volume", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/vagrant/ceph-volume/bin/ceph-volume", line 6, in <module>
    main.Volume()
  File "/home/vagrant/ceph-volume/ceph_volume/main.py", line 40, in __init__
    self.main(self.argv)
  File "/home/vagrant/ceph-volume/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/home/vagrant/ceph-volume/ceph_volume/main.py", line 156, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/home/vagrant/ceph-volume/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/home/vagrant/ceph-volume/ceph_volume/devices/lvm/main.py", line 40, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/home/vagrant/ceph-volume/ceph_volume/terminal.py", line 182, in dispatch
    instance.main()
  File "/home/vagrant/ceph-volume/ceph_volume/devices/lvm/zap.py", line 177, in main
    self.zap(args)
  File "/home/vagrant/ceph-volume/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/home/vagrant/ceph-volume/ceph_volume/devices/lvm/zap.py", line 78, in zap
    osd_path = "/var/lib/ceph/osd/{}-{}".format(lv.tags['ceph.cluster_name'], lv.tags['ceph.osd_id'])
KeyError: 'ceph.cluster_name'

#4 Updated by Alfredo Deza 11 months ago

  • Status changed from Verified to Resolved
  • Assignee set to Andrew Schoen

Also available in: Atom PDF