Bug #58946
closedcephadm: KeyError: 'osdspec_affinity'
0%
Description
/a/yuriw-2023-03-08_22:50:12-rados-wip-yuri4-testing-2023-03-08-1234-distro-default-smithi/7198692
2023-03-09T07:14:42.033 INFO:journalctl@ceph.mon.a.smithi182.stdout:Mar 09 07:14:41 smithi182 ceph-mon[105303]: Failed to apply osd.dashboard-admin-1678346078356 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout: service_id: dashboard-admin-1678346078356
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout: service_name: osd.dashboard-admin-1678346078356
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout: placement:
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout: host_pattern: '*'
2023-03-09T07:14:42.034 INFO:journalctl@ceph.mon.a.smithi182.stdout: spec:
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: data_devices:
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: rotational: true
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: filter_logic: AND
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: objectstore: bluestore
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: ''')): 'osdspec_affinity'
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: Traceback (most recent call last):
2023-03-09T07:14:42.035 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 545, in _apply_all_services
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: if self._apply_service(spec):
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 660, in _apply_service
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: ret = self.mgr.wait_async(all_hosts())
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 690, in wait_async
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: return self.event_loop.get_result(coro)
2023-03-09T07:14:42.036 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: return self.__get_result()
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: raise self._exception
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 74, in all_hosts
2023-03-09T07:14:42.037 INFO:journalctl@ceph.mon.a.smithi182.stdout: for h, ds in self.prepare_drivegroup(drive_group)]
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 229, in prepare_drivegroup
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: existing_daemons=len(dd_for_spec_and_host))
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: self._data = self.assign_devices('data_devices', self.spec.data_devices)
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: return f(self, ds)
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices
2023-03-09T07:14:42.038 INFO:journalctl@ceph.mon.a.smithi182.stdout: if lv['osdspec_affinity'] != self.spec.service_id:
2023-03-09T07:14:42.039 INFO:journalctl@ceph.mon.a.smithi182.stdout: KeyError: 'osdspec_affinity'
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-08_22:50:12-rados-wip-yuri4-testing-2023-03-08-1234-distro-default-smithi/7198692/remote/smithi182/log/afa27f4c-be48-11ed-9aec-001a4aab830c/ceph-mgr.a.log.gz
2023-03-09T07:14:40.082+0000 7feee409d700 0 [cephadm DEBUG cephadm.services.osd] Found inventory for host [Device(path=/dev/nvme0n1, lvs=[{'comment': 'not used by ceph', 'name': 'lv_1'}, {'block_uuid': 'Xu33Ig-UbcL-21yk-KcjN-2naN-WtTj-Uekovc', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_2', 'osd_fsid': 'db5c95f7-961f-4eee-999e-67531e915ccb', 'osd_id': '2', 'osdspec_affinity': 'None', 'type': 'block'}, {'block_uuid': 'rMh4U3-RVi1-TTRN-WtTs-fxWg-GbaZ-R4E7ON', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_3', 'osd_fsid': 'f53fa360-6a84-48fd-a256-437a5d177882', 'osd_id': '1', 'osdspec_affinity': 'None', 'type': 'block'}, {'block_uuid': 'kzcFnC-vtcI-iDat-obgv-p65k-83GL-CdQPU1', 'cluster_fsid': 'afa27f4c-be48-11ed-9aec-001a4aab830c', 'cluster_name': 'ceph', 'name': 'lv_4', 'osd_fsid': '95a61b7e-e672-45fc-914b-2b7798e2a34b', 'osd_id': '0', 'osdspec_affinity': 'None', 'type': 'block'}, {'comment': 'not used by ceph', 'name': 'lv_5'}], available=False, ceph_device=True, crush_device_class=None, rejection reasons=['LVM detected', 'locked'])]
2023-03-09T07:14:40.082+0000 7feee409d700 0 [cephadm DEBUG ceph.deployment.drive_selection.selector] Processing disk /dev/nvme0n1
2023-03-09T07:14:40.084+0000 7feee60a1700 0 [cephadm ERROR cephadm.serve] Failed to apply osd.dashboard-admin-1678346078356 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
service_id: dashboard-admin-1678346078356
service_name: osd.dashboard-admin-1678346078356
placement:
host_pattern: '*'
spec:
data_devices:
rotational: true
filter_logic: AND
objectstore: bluestore
''')): 'osdspec_affinity'
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 545, in _apply_all_services
if self._apply_service(spec):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 660, in _apply_service
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 77, in create_from_spec
ret = self.mgr.wait_async(all_hosts())
File "/usr/share/ceph/mgr/cephadm/module.py", line 690, in wait_async
return self.event_loop.get_result(coro)
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result
return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 74, in all_hosts
for h, ds in self.prepare_drivegroup(drive_group)]
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 229, in prepare_drivegroup
existing_daemons=len(dd_for_spec_and_host))
File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 35, in __init__
self._data = self.assign_devices('data_devices', self.spec.data_devices)
File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 19, in wrapper
return f(self, ds)
File "/lib/python3.6/site-packages/ceph/deployment/drive_selection/selector.py", line 134, in assign_devices
if lv['osdspec_affinity'] != self.spec.service_id:
KeyError: 'osdspec_affinity'
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-09_16:25:32-rados-wip-yuri11-testing-2023-03-08-1220-distro-default-smithi/7199848
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-08_15:52:33-rados-wip-yuri8-testing-2023-03-07-1527-distro-default-smithi/7197749
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-10_22:37:27-rados-wip-yuri7-testing-2023-03-10-0830-distro-default-smithi/7202748
Updated by Laura Flores about 1 year ago
/a/lflores-2023-03-27_02:17:31-rados-wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a-distro-default-smithi/7221076
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-27_23:05:54-rados-wip-yuri4-testing-2023-03-25-0714-distro-default-smithi/7222026
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-30_21:53:20-rados-wip-yuri7-testing-2023-03-29-1100-distro-default-smithi/7228047
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7224179
Updated by Laura Flores about 1 year ago
/a/yuriw-2023-03-30_21:29:24-rados-wip-yuri2-testing-2023-03-30-0826-distro-default-smithi/7227529
Updated by Laura Flores 12 months ago
/a/yuriw-2023-04-25_14:15:40-rados-pacific-release-distro-default-smithi/7251198
Updated by Sridhar Seshasayee 12 months ago
/a/sseshasa-2023-05-02_03:09:13-rados-wip-sseshasa-testing-2023-05-01-2145-distro-default-smithi/7260260
Updated by Laura Flores 12 months ago
/a/yuriw-2023-04-25_21:30:50-rados-wip-yuri3-testing-2023-04-25-1147-distro-default-smithi/7253515
Updated by Laura Flores 12 months ago
- Project changed from Orchestrator to Dashboard
Discussed this with Adam King, looks like an issue with this particular dashboard test.
Updated by Laura Flores 12 months ago
/a/yuriw-2023-04-24_22:54:45-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7250392
Updated by Laura Flores 11 months ago
/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276311
Updated by Laura Flores 11 months ago
/a/yuriw-2023-05-11_15:01:38-rados-wip-yuri8-testing-2023-05-10-1402-distro-default-smithi/7271334
Updated by Laura Flores 11 months ago
- Assignee set to Nizamudeen A
Hey Nizam, would you mind checking this to see if it's a problem in the dashboard?
/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270708
Updated by Laura Flores 11 months ago
/a/yuriw-2023-05-19_19:19:25-rados-wip-yuri11-testing-2023-05-19-0836-distro-default-smithi/7279319
Updated by Laura Flores 11 months ago
- Backport changed from pacific to pacific,quincy,reef
/a/yuriw-2023-05-20_14:42:29-rados-wip-yuri6-testing-2023-05-19-1351-reef-distro-default-smithi/7281115
Updated by Laura Flores 11 months ago
/a/yuriw-2023-05-22_23:22:00-rados-wip-yuri-testing-2023-05-22-0845-reef-distro-default-smithi/7283110
Updated by Laura Flores 11 months ago
/a/yuriw-2023-05-24_14:33:21-rados-wip-yuri6-testing-2023-05-23-0757-reef-distro-default-smithi/7285190
Updated by Laura Flores 11 months ago
/a/yuriw-2023-06-13_14:58:18-rados-wip-yuri8-testing-2023-06-12-1236-reef-distro-default-smithi/7302615
Updated by Laura Flores 10 months ago
/a/yuriw-2023-06-22_20:29:56-rados-wip-yuri3-testing-2023-06-22-0812-reef-distro-default-smithi/7313225
Updated by Laura Flores 10 months ago
/a/yuriw-2023-06-23_20:51:14-rados-wip-yuri8-testing-2023-06-22-1309-pacific-distro-default-smithi/7314195
Updated by Laura Flores 10 months ago
/a/yuriw-2023-07-03_15:32:36-rados-wip-yuri5-testing-2023-06-28-1515-distro-default-smithi/7325585
Updated by Laura Flores 9 months ago
/a/yuriw-2023-07-17_14:37:31-rados-wip-yuri-testing-2023-07-14-1641-distro-default-smithi/7341505
Updated by Laura Flores 9 months ago
- Project changed from Dashboard to Orchestrator
Moving to the Orch project. Perhaps it's a cephadm problem?
Updated by Laura Flores 9 months ago
Seems like it was introduced in https://github.com/ceph/ceph/pull/49969.
Updated by Laura Flores 9 months ago
- Assignee changed from Nizamudeen A to Guillaume Abrioux
Updated by Guillaume Abrioux 9 months ago
- Status changed from New to In Progress
Updated by Matan Breizman 9 months ago
/a/yuriw-2023-07-18_14:02:17-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342910
/a/yuriw-2023-07-18_14:02:17-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342961
Updated by Guillaume Abrioux 9 months ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 52532
Updated by Adam Huffman 8 months ago
Just noting that it appears we're suffering from this on a 16.2.13 cluster, as noted on the mailing list.
Updated by Laura Flores 8 months ago
- Status changed from Fix Under Review to Pending Backport
Updated by Laura Flores 8 months ago
- Copied to Backport #62590: reef: cephadm: KeyError: 'osdspec_affinity' added
Updated by Laura Flores 8 months ago
- Copied to Backport #62591: pacific: cephadm: KeyError: 'osdspec_affinity' added
Updated by Laura Flores 8 months ago
- Copied to Backport #62592: quincy: cephadm: KeyError: 'osdspec_affinity' added