Ceph » Orchestrator

Certain failures of commands that we run asynchronously using asyncio to provide a timeout feature can raise a concurrent.futures.CancelledError upon failure. The handler we have for the asyncio timeout stuff doesn't handle this type of exception and the cephadm module ends up crashing. This might only be happening on builds using a python version later than 3.6, so we haven't seen it reported much. Have seen it in python 3.9 builds.

2024-02-15T14:14:00.147593+0000 mgr.cephnode1.rfvatv (mgr.16779) 138077 : cephadm [ERR] executing refresh((['cephnode1', 'cephnode2', 'cephnode3', 'cephnode4'],)) failed.
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/utils.py", line 94, in do_work
    return f(*arg)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 267, in refresh
    r = self._refresh_facts(host)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 370, in _refresh_facts
    val = self.mgr.wait_async(self._run_cephadm_json(
  File "/usr/share/ceph/mgr/cephadm/module.py", line 671, in wait_async
    return self.event_loop.get_result(coro, timeout)
  File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result
    return future.result(timeout)
  File "/lib64/python3.9/concurrent/futures/_base.py", line 444, in result
    raise CancelledError()
concurrent.futures._base.CancelledError