Project

General

Profile

Bug #58137

OSD prepare job fails with KeyError: 'KNAME'

Added by Guillaume Abrioux 2 months ago. Updated 13 days ago.

Status:
Pending Backport
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[2022-11-13 18:27:04,778][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
    tmpfs,
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 51, in prepare_bluestore
    block = prepare_dmcrypt(key, block, 'block', fsid)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 23, in prepare_dmcrypt
    kname = disk.lsblk(device)['KNAME']
KeyError: 'KNAME'
[2022-11-13 18:27:04,780][ceph_volume.devices.raw.prepare][INFO  ] will rollback OSD ID creation
[2022-11-13 18:27:04,781][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.4 --yes-i-really-mean-it
[2022-11-13 18:27:05,553][ceph_volume.process][INFO  ] stderr purged osd.4
[2022-11-13 18:27:05,571][ceph_volume.process][INFO  ] Running command: /usr/bin/systemctl is-active ceph-osd@4
[2022-11-13 18:27:05,584][ceph_volume.process][INFO  ] stderr System has not been booted with systemd as init system (PID 1). Can't operate.
[2022-11-13 18:27:05,585][ceph_volume.process][INFO  ] stderr Failed to connect to bus: Host is down
[2022-11-13 18:27:05,589][ceph_volume.util.system][WARNING] Executable lvs not found on the host, will return lvs as-is
[2022-11-13 18:27:05,590][ceph_volume.process][INFO  ] Running command: lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix -S tags={ceph.osd_id=4} -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2022-11-13 18:27:05,969][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
    tmpfs,
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 51, in prepare_bluestore
    block = prepare_dmcrypt(key, block, 'block', fsid)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 23, in prepare_dmcrypt
    kname = disk.lsblk(device)['KNAME']
KeyError: 'KNAME'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
    self.safe_prepare(self.args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 95, in safe_prepare
    rollback_osd(self.args, self.osd_id)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/common.py", line 35, in rollback_osd
    Zap(['--destroy', '--osd-id', osd_id]).main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 404, in main
    self.zap_osd()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 301, in zap_osd
    devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices
    '%s' % osd_id or osd_fsid)
RuntimeError: Unable to find any LV for zapping OSD: 4

see https://github.com/rook/rook/issues/11304 for details


Related issues

Copied to ceph-volume - Backport #58574: quincy: OSD prepare job fails with KeyError: 'KNAME' In Progress
Copied to ceph-volume - Backport #58575: pacific: OSD prepare job fails with KeyError: 'KNAME' In Progress

History

#1 Updated by Guillaume Abrioux 2 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 49171

#2 Updated by Guillaume Abrioux 13 days ago

  • Status changed from Fix Under Review to Pending Backport

#3 Updated by Backport Bot 13 days ago

  • Copied to Backport #58574: quincy: OSD prepare job fails with KeyError: 'KNAME' added

#4 Updated by Backport Bot 13 days ago

  • Copied to Backport #58575: pacific: OSD prepare job fails with KeyError: 'KNAME' added

#5 Updated by Backport Bot 13 days ago

  • Tags set to backport_processed

Also available in: Atom PDF