Ceph » Orchestrator

parallelism of the ssh orchestrator is not trivial:

https://github.com/ceph/ceph/blob/713db2994dd262a687612783c85f7929e9041d9c/src/pybind/mgr/ssh/module.py#L272

sets the thread pool size to 1, which is really suboptimal. Changing this requires something like Lock to prevent conflicts

But this is still not really sufficient, if we have a big cluster, as we cannot increase this limit indefinitely as we're talking about kernel threads here. We can improve this by using something like https://pypi.org/project/parallel-ssh/ instead of remoto.

Regarding concurrency, there is a open PR ceph/ceph: Pull Request 26565 to allow executing multiple commands concurrent

So, these are the options we have right now:

| What                                   | Effort | Result |
|----------------------------------------|--------|---------------------|
| increase thread pool size to 10        | small  | limited improvement |
| https://pypi.org/project/parallel-ssh/ | big    | big improvement     |
| pyzmq instead of SSH                   | huge   | everything parallel |