Ceph » Orchestrator

If an exception is thrown while cephadm is attempting to apply a service spec in the serve loop, the progress indicator for the failed deployment gets stuck. For example, we had stuck progress indicators after cephadm encountered exceptions while attempting to remove a couple of crash daemons. The following was our output from the ceph status command:

root@ceph01:~# ceph -s
  cluster:
    id:     <redacted>
    health: HEALTH_OK
  services:
    mon: 5 daemons, quorum ceph01,ceph02,ceph03,ceph04,ceph05 (age 58m)
    mgr: ceph01.sqycox(active, since 25m), standbys: ceph02.zmwnqy, ceph03.rtpqra, ceph05.jsdbre, ceph04.qkxmtx
    osd: 504 osds: 504 up (since 13m), 504 in (since 5h)
  data:
    pools:   3 pools, 5121 pgs
    objects: 6.87k objects, 23 GiB
    usage:   31 TiB used, 6.3 PiB / 6.3 PiB avail
    pgs:     5121 active+clean
  io:
    client:   2.8 KiB/s wr, 0 op/s rd, 0 op/s wr
  progress:
    Updating crash deployment (-2 -> 19) (0s)
      [............................]
    Updating crash deployment (-2 -> 19) (0s)
      [............................]

Since progress data is stored in memory on the active manager, the stuck progress indicators can be cleared by restarting the active manager.

Relates to the issue described in: https://tracker.ceph.com/issues/51616