Ceph » Orchestrator

It looks like where (which host) services are running on is not correctly considered when ceph orch host maintenance enter A is requested. It looks like it simply considers the count of running services and whether the host should have a service running, ignoring the fact that the service could be in "error" mode on the host and actually it was running on another host.

1. Deploy service X (say, MGR) on 2 hosts: A and B.
2. Get X to error state on A - it could be running but cephadm just doesn't know that it is running, for example cluster could be in state CEPHADM_FAILED_DAEMON (this should be a subject for a separate bug report, but ignore the specifics for now). What's important is that X is running on one host and we want to have the other host in maintenance mode.
3. X is now running "1/2", with cephadm knowing it definitely is OK on B, for example:
   

   ceph orch ls|grep mgr
   NAME              PORTS                   RUNNING  REFRESHED  AGE  PLACEMENT
   mgr                                           1/2  36s ago    3M   count:2
   
   ceph orch ps|grep mgr
   mgr.A                A    *:9283       error           105s ago   3D    77.9M        -  17.2.3
   mgr.B                B    *:9283       running (3d)    105s ago   3D    89.9M        -  17.2.3
   

1. Try to enter maintenance mode for node A.
2. cephadm will complain that there would be zero MGR running after maintenance mode. But it is running on node B, it is not currently running on A! In this example both MGR and Grafana services were running on B and were in error state on A:
   

   ALERT: Cannot stop ['mgr.A.borked'] in Mgr service. Not enough remaining Mgr daemons. Please deploy at least 2 Mgr daemons before stopping ['mgr.A.borked'].
   WARNING: Stopping 1 out of 1 daemons in Grafana service. Service will not be operational with no daemons left. At least 1 daemon must be running to guarantee service.
   

It looks like there is some naive algorithm like "IF service should run on this host AND servicecount - 1 IS not ok, THEN deny maintenance".
While the correct way would be "IF service should run on this host AND really is currently running on this host AND servicecount - 1 is not ok, THEN deny maintenance".

I understand there could be much more to the algorithm but the end result is the same, it doesn't work right. cephadm doesn't correctly consider services in "error" state on a machine that has maintenance mode requested and it needs to take the "error" state into account.