Ceph » Orchestrator

After bootstrap is completed, one needs to add the generated ceph ssh keys to the concerned hosts before adding them to the cluster using command

ceph orch host add

The issue was initially observed with a Crimson image, but was later found to a be underlying Reef issue
https://tracker.ceph.com/issues/58918 -> Currently part of mgr project

Steps:
Completed cephadm bootstrap
Copy ssh keys to designated hosts expect for one
Get into cephadm shell and all the hosts to the cluster
All the hosts will get added to cluster for which the ssh keys were copied

The hosts to which ssh keys have not been copied will throw this exception when they are added to the cluster -

Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 1761, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command
    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda>
    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # noqa: E731
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 453, in _add_host
    return self._apply_misc([s], False, Format.plain)
  File "/usr/share/ceph/mgr/orchestrator/module.py", line 1246, in _apply_misc
    raise_if_exception(completion)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception
    e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr'

The expected error message in such a scenario as observed from previous releases should be about ssh connection failing instead of hostname and addr missing -

# ceph orch host add ceph-hakumar-8t8ruh-node4 10.0.210.10 node4-label
Error EINVAL: Failed to connect to ceph-hakumar-8t8ruh-node4 (10.0.210.10). Permission denied
Log: Opening SSH connection to 10.0.210.10, port 22
[conn=4, chan=1415] Received exit status 0
[conn=4, chan=1415] Received channel close
[conn=4, chan=1415] Channel closed
[conn=5] Connected to SSH server at 10.0.210.10, port 22
[conn=5]   Local address: 10.0.208.106, port 44566
[conn=5]   Peer address: 10.0.210.10, port 22
[conn=5] Beginning auth for user root
[conn=5] Auth failed for user root
[conn=5] Connection failure: Permission denied
[conn=5] Aborting connection