Ceph » mgr » Dashboard

Description of problem¶

While upgrading from 16.2.7 to 16.2.8 (but likely unrelated to this new release, since no iSCSI changes have been made there):

xxxxxxxxxxxxxxxxxxxx mgr.ceph-01.xxxxx [WRN] Failed to set Dashboard config for iSCSI: dashboard iscsi-gateway-add failed: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1448, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 414, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/mgr_module.py", line 450, in check
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_cli.py", line 36, in add_iscsi_gateway
    IscsiGatewaysConfig.add_gateway(name, service_url)
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 87, in add_gateway
    config = cls.get_gateways_config()
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 105, in get_gateways_config
    return cls._load_config_from_store()
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 48, in _load_config_from_store
    cls.update_iscsi_config(config)
  File "/usr/share/ceph/mgr/dashboard/services/iscsi_config.py", line 58, in update_iscsi_config
    for gateway_name, gateway_config in config['gateways'].items():
RuntimeError: dictionary changed size during iteration
 retval: -22

The user reported that their iSCSI deployment didn't exhibit any issue though.

dictionary changed size during iteration exception is raised when a dictionary is mutated while iterated:

• it can happen because the dictionary is modified in the very loop where it's iterated,
• or when multiple threads are accessing the same dictionary.