Bug #37514
closedmgr CLI commands block one another (indefinitely if the orchestrator CLI gets stuck in wait)
0%
Description
There seems to be two problems here:
1) Only one mgr CLI command runs at a time. This isn't obvious unless you find an mgr command that takes a few seconds to run. For example, when testing the deepsea orchestrator, `ceph orchestrator device ls` takes about five seconds to complete. If I invoke that command in one terminal window, then invoke `ceph osd status` in another terminal window, the latter will block until the former completes. That's irritating, but probably not fatal.
2) Orchestrator CLI commands spin waiting for command completions from whatever orchestrator module is active. If you manage to break an orchestrator in such a way as commands never complete (try e.g.: stopping the salt-api while using DeepSea), `ceph orchestrator device ls` will never complete. Even if you hit CTRL-C, mgr is (presumably) still stuck in that loop waiting for completions that are never going to happen, which means it's unable to service any subsequent CLI command. Trying to run, say, `ceph osd status` at this point will also simply just hang. You can't even quickly restart mgr at this point: `systemctl restart ceph-mgr$(hostname)`@ hangs until it hits "State 'stop-sigterm' timed out." (after maybe a minute and a half), then it sends a SIGKILL and mgr is finally restarted.