Project

General

Profile

Bug #53351

mgr/cephadm: when the cephadm agent refreshes the mgr host case, the config checker can throw RuntimeError: dictionary changed size during iteration

Added by Paul Cuzner over 2 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
cephadm
Target version:
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When cephadm agent is in play, the mgr cache can be updated at any point, causing a runtime error in the config checkers process+hosts function.
e.g.

2021-11-19T22:26:11.811+0000 7f6101785700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.service-02-08039.gyaydv: dictionary cha
nged size during iteration
2021-11-19T22:26:11.811+0000 7f6101785700 -1 cephadm.serve:
2021-11-19T22:26:11.811+0000 7f6101785700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/module.py", line 543, in serve
    serve.serve()
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 69, in serve
    self._refresh_hosts_and_daemons()
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 346, in _refresh_hosts_and_daemons
    self.mgr.config_checker.run_checks()
  File "/usr/share/ceph/mgr/cephadm/configchecks.py", line 693, in run_checks
    self._process_hosts()
  File "/usr/share/ceph/mgr/cephadm/configchecks.py", line 639, in _process_hosts
    for hostname in self.mgr.cache.facts:
RuntimeError: dictionary changed size during iteration

History

#1 Updated by Paul Cuzner over 2 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 44037

#2 Updated by Redouane Kachach Elhichou almost 2 years ago

  • Status changed from Fix Under Review to New
  • Pull request ID deleted (44037)

Also available in: Atom PDF