Project

General

Profile

Actions

Bug #58946

closed

cephadm: KeyError: 'osdspec_affinity'

Added by Laura Flores about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-03-08_22:50:12-rados-wip-yuri4-testing-2023-03-08-1234-distro-default-smithi/7198692

2023-03-09T07:14:42.033 INFO:journalctl@ceph.mon.a.smithi182.stdout:Mar 09 07:14:41 smithi182 ceph-mon[105303]: Failed to apply osd.dashboard-admin-1678346078356 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            service_id: dashboard-admin-1678346078356
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            service_name: osd.dashboard-admin-1678346078356
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            placement:
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              host_pattern: '*'
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            spec:
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              data_devices:
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                rotational: true
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              filter_logic: AND
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              objectstore: bluestore
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            ''')): 'osdspec_affinity'
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            Traceback (most recent call last):
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/serve.py", line 545, in _apply_all_services
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                if self._apply_service(spec):
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/serve.py", line 660, in _apply_service
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                ret = self.mgr.wait_async(all_hosts())
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/module.py", line 690, in wait_async
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                return self.event_loop.get_result(coro)
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                return self.__get_result()
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                raise self._exception
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 74, in all_hosts
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                for h, ds in self.prepare_drivegroup(drive_group)]
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 229, in prepare_drivegroup
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                existing_daemons=len(dd_for_spec_and_host))
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                self._data = self.assign_devices('data_devices', self.spec.data_devices)
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                return f(self, ds)
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                              File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                                if lv['osdspec_affinity'] != self.spec.service_id:
2023-03-09T07:14:42.039 INFO:journalctl@ceph.mon.a.smithi182.stdout:                                            KeyError: 'osdspec_affinity'


Related issues 3 (0 open3 closed)

Copied to Orchestrator - Backport #62590: reef: cephadm: KeyError: 'osdspec_affinity'ResolvedLaura FloresActions
Copied to Orchestrator - Backport #62591: pacific: cephadm: KeyError: 'osdspec_affinity'ResolvedLaura FloresActions
Copied to Orchestrator - Backport #62592: quincy: cephadm: KeyError: 'osdspec_affinity'ResolvedLaura FloresActions
Actions #1

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-08_22:50:12-rados-wip-yuri4-testing-2023-03-08-1234-distro-default-smithi/7198692/remote/smithi182/log/afa27f4c-be48-11ed-9aec-001a4aab830c/ceph-mgr.a.log.gz

2023-03-09T07:14:40.082+0000 7feee409d700  0 [cephadm DEBUG cephadm.services.osd] Found inventory for host [Device(path=/dev/nvme0n1, lvs=[{'comment': 'not used by ceph', 'name': 'lv_1'}, {'block_uuid': 'Xu33Ig-UbcL-21yk-KcjN-2naN-WtTj-Uekovc', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_2', 'osd_fsid': 'db5c95f7-961f-4eee-999e-67531e915ccb', 'osd_id': '2', 'osdspec_affinity': 'None', 'type': 'block'}, {'block_uuid': 'rMh4U3-RVi1-TTRN-WtTs-fxWg-GbaZ-R4E7ON', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_3', 'osd_fsid': 'f53fa360-6a84-48fd-a256-437a5d177882', 'osd_id': '1', 'osdspec_affinity': 'None', 'type': 'block'}, {'block_uuid': 'kzcFnC-vtcI-iDat-obgv-p65k-83GL-CdQPU1', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_4', 'osd_fsid': '95a61b7e-e672-45fc-914b-2b7798e2a34b', 'osd_id': '0', 'osdspec_affinity': 'None', 'type': 'block'}, {'comment': 'not used by ceph', 'name': 'lv_5'}], available=False, ceph_device=True, crush_device_class=None, rejection reasons=['LVM detected', 'locked'])]
2023-03-09T07:14:40.082+0000 7feee409d700  0 [cephadm DEBUG ceph.deployment.drive_selection.selector] Processing disk /dev/nvme0n1
2023-03-09T07:14:40.084+0000 7feee60a1700  0 [cephadm ERROR cephadm.serve] Failed to apply osd.dashboard-admin-1678346078356 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
service_id: dashboard-admin-1678346078356
service_name: osd.dashboard-admin-1678346078356
placement:
  host_pattern: '*'
spec:
  data_devices:
    rotational: true
  filter_logic: AND
  objectstore: bluestore
''')): 'osdspec_affinity'
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 545, in _apply_all_services
    if self._apply_service(spec):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 660, in _apply_service
    self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec
    ret = self.mgr.wait_async(all_hosts())
  File "/usr/share/ceph/mgr/cephadm/module.py", line 690, in wait_async
    return self.event_loop.get_result(coro)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result
    return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
  File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 74, in all_hosts
    for h, ds in self.prepare_drivegroup(drive_group)]
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 229, in prepare_drivegroup
    existing_daemons=len(dd_for_spec_and_host))
  File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__
    self._data = self.assign_devices('data_devices', self.spec.data_devices)
  File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper
    return f(self, ds)
  File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices
    if lv['osdspec_affinity'] != self.spec.service_id:
KeyError: 'osdspec_affinity'

Actions #2

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-09_16:25:32-rados-wip-yuri11-testing-2023-03-08-1220-distro-default-smithi/7199848

Actions #3

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-08_15:52:33-rados-wip-yuri8-testing-2023-03-07-1527-distro-default-smithi/7197749

Actions #4

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-10_22:37:27-rados-wip-yuri7-testing-2023-03-10-0830-distro-default-smithi/7202748

Actions #5

Updated by Laura Flores about 1 year ago

/a/lflores-2023-03-27_02:17:31-rados-wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a-distro-default-smithi/7221076

Actions #6

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-27_23:05:54-rados-wip-yuri4-testing-2023-03-25-0714-distro-default-smithi/7222026

Actions #7

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-30_21:53:20-rados-wip-yuri7-testing-2023-03-29-1100-distro-default-smithi/7228047

Actions #8

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7224179

Actions #9

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-30_21:29:24-rados-wip-yuri2-testing-2023-03-30-0826-distro-default-smithi/7227529

Actions #10

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-25_14:15:40-rados-pacific-release-distro-default-smithi/7251198

Actions #11

Updated by Laura Flores 12 months ago

  • Backport set to pacific
Actions #12

Updated by Sridhar Seshasayee 12 months ago

/a/sseshasa-2023-05-02_03:09:13-rados-wip-sseshasa-testing-2023-05-01-2145-distro-default-smithi/7260260

Actions #13

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-25_21:30:50-rados-wip-yuri3-testing-2023-04-25-1147-distro-default-smithi/7253515

Actions #14

Updated by Laura Flores 12 months ago

  • Project changed from Orchestrator to Dashboard

Discussed this with Adam King, looks like an issue with this particular dashboard test.

Actions #15

Updated by Laura Flores 11 months ago

/a/yuriw-2023-04-24_22:54:45-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7250392

Actions #16

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276311

Actions #17

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-11_15:01:38-rados-wip-yuri8-testing-2023-05-10-1402-distro-default-smithi/7271334

Actions #18

Updated by Laura Flores 11 months ago

  • Assignee set to Nizamudeen A

Hey Nizam, would you mind checking this to see if it's a problem in the dashboard?

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270708

Actions #19

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-19_19:19:25-rados-wip-yuri11-testing-2023-05-19-0836-distro-default-smithi/7279319

Actions #20

Updated by Laura Flores 11 months ago

  • Backport changed from pacific to pacific,quincy,reef

/a/yuriw-2023-05-20_14:42:29-rados-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7281115

Actions #21

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-22_23:22:00-rados-wip-yuri-testing-2023-05-22-0845-reef-distro-default-smithi/7283110

Actions #22

Updated by Laura Flores 11 months ago

/a/yuriw-2023-05-24_14:33:21-rados-wip-yuri6-testing-2023-05-23-0757-reef-distro-default-smithi/7285190

Actions #23

Updated by Laura Flores 10 months ago

/a/yuriw-2023-06-13_14:58:18-rados-wip-yuri8-testing-2023-06-12-1236-reef-distro-default-smithi/7302615

Actions #24

Updated by Laura Flores 10 months ago

  • Priority changed from Normal to High
Actions #25

Updated by Laura Flores 10 months ago

/a/yuriw-2023-06-22_20:29:56-rados-wip-yuri3-testing-2023-06-22-0812-reef-distro-default-smithi/7313225

Actions #26

Updated by Laura Flores 9 months ago

/a/yuriw-2023-06-23_20:51:14-rados-wip-yuri8-testing-2023-06-22-1309-pacific-distro-default-smithi/7314195

Actions #27

Updated by Laura Flores 9 months ago

/a/yuriw-2023-07-03_15:32:36-rados-wip-yuri5-testing-2023-06-28-1515-distro-default-smithi/7325585

Actions #28

Updated by Laura Flores 9 months ago

/a/yuriw-2023-07-17_14:37:31-rados-wip-yuri-testing-2023-07-14-1641-distro-default-smithi/7341505

Actions #29

Updated by Laura Flores 9 months ago

  • Project changed from Dashboard to Orchestrator

Moving to the Orch project. Perhaps it's a cephadm problem?

Actions #30

Updated by Laura Flores 9 months ago

Seems like it was introduced in https://github.com/ceph/ceph/pull/49969.

Actions #31

Updated by Laura Flores 9 months ago

  • Assignee changed from Nizamudeen A to Guillaume Abrioux
Actions #32

Updated by Guillaume Abrioux 9 months ago

  • Status changed from New to In Progress
Actions #33

Updated by Matan Breizman 9 months ago

/a/yuriw-2023-07-18_14:02:17-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342910
/a/yuriw-2023-07-18_14:02:17-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342961

Actions #34

Updated by Guillaume Abrioux 9 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 52532
Actions #35

Updated by Adam Huffman 8 months ago

Just noting that it appears we're suffering from this on a 16.2.13 cluster, as noted on the mailing list.

Actions #36

Updated by Laura Flores 8 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions #37

Updated by Laura Flores 8 months ago

  • Copied to Backport #62590: reef: cephadm: KeyError: 'osdspec_affinity' added
Actions #38

Updated by Laura Flores 8 months ago

  • Copied to Backport #62591: pacific: cephadm: KeyError: 'osdspec_affinity' added
Actions #39

Updated by Laura Flores 8 months ago

  • Copied to Backport #62592: quincy: cephadm: KeyError: 'osdspec_affinity' added
Actions #40

Updated by Laura Flores 8 months ago

  • Tags set to backport_processed
Actions #41

Updated by Adam King 6 months ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF