Bug #44820: racey concurrent ceph-volume callKeyError: 'ceph.type' - Orchestrator - Ceph

Actions

Copy link

Bug #44820

closed

racey concurrent ceph-volume callKeyError: 'ceph.type'

Added by Sage Weil about 4 years ago. Updated about 4 years ago.

Status:

Resolved

Priority:

High

Assignee:

Category:

Target version:

Ceph - v15.2.2

% Done:

Source:

Tags:

Backport:

octopus

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

34320

Crash signature (v1):

Crash signature (v2):

Description

[2020-03-30 20:13:57,357][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 150, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 38, in main
    self.format_report(Devices())
  File "/usr/lib/python3.6/site-packages/ceph_volume/inventory/main.py", line 42, in format_report
    print(json.dumps(inventory.json_report()))
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 51, in json_report
    output.append(device.json_report())
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 202, in json_report
    output['lvs'] = [lv.report() for lv in self.lvs]
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 202, in <listcomp>
    output['lvs'] = [lv.report() for lv in self.lvs]
  File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 941, in report
    type_ = self.tags['ceph.type']
KeyError: 'ceph.type'

/a/sage-2020-03-30_19:43:15-rados:cephadm:smoke-roleless-wip-sage-testing-2020-03-30-0849-distro-basic-smithi/4908307
/a/sage-2020-03-30_19:43:15-rados:cephadm:smoke-roleless-wip-sage-testing-2020-03-30-0849-distro-basic-smithi/4908319

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Sage Weil about 4 years ago

the problem seems to be a racing invocation of inventory and prepare:

2020-03-30T20:13:53.212+0000 7ffadb569700  0 [cephadm DEBUG root] args: --image quay.io/ceph-ci/ceph:7b6c4e4780f8e032fea535516abbc2b9e7667b72 ceph-volume --fsid 81059e96-72c2-11ea-924b-001a4aab830c --config-json - -- lvm prepare --bluestore --data vg_nvme/lv_3 --no-systemd
...
2020-03-30T20:13:53.944+0000 7ffacf618700  0 [cephadm DEBUG root] args: --image quay.io/ceph-ci/ceph:7b6c4e4780f8e032fea535516abbc2b9e7667b72 ceph-volume --fsid 81059e96-72c2-11ea-924b-001a4aab830c -- inventory --format=json
...
RuntimeError: Failed command: /bin/podman run --rm --net=host --privileged --group-add=disk -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:7b6c4e4780f8e032fea535516abbc2b9e7667b72 -e NODE_NAME=smithi192 -v /var/run/ceph/81059e96-72c2-11ea-924b-001a4aab830c:/var/run/ceph:z -v /var/log/ceph/81059e96-72c2-11ea-924b-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/81059e96-72c2-11ea-924b-001a4aab830c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm --entrypoint /usr/sbin/ceph-volume quay.io/ceph-ci/ceph:7b6c4e4780f8e032fea535516abbc2b9e7667b72 inventory --format=json
2020-03-30T20:13:58.063+0000 7ffacf618700  0 [cephadm ERROR root] cephadm exited with an error code: 1, stderr:INFO:cephadm:/bin/podman:stderr -->  KeyError: 'ceph.type'

Actions

Copy link

Updated by Sage Weil about 4 years ago

Status changed from New to Fix Under Review
Pull request ID set to 34320

Actions

Copy link

Updated by Sage Weil about 4 years ago

Backport set to octopus

Actions

Copy link

Updated by Jan Fajerski about 4 years ago

Copied to Bug #44852: KeyError: 'ceph.type' added

Actions

Copy link

Updated by Jan Fajerski about 4 years ago

Project changed from ceph-volume to Orchestrator
Subject changed from KeyError: 'ceph.type' to racey concurrent ceph-volume callKeyError: 'ceph.type'
Severity deleted (~~3 - minor~~)

Copying to fix in c-v as well.

Actions

Copy link