Ceph » Orchestrator

The staggered upgrade test check the "ceph versions" output in order to verify it has upgraded only the expected set of daemons. For example part way through the test we try to upgrade only a subset of the mon daemons, which gets"ceph versions" output like

{
    "mon": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2,
        "ceph version 18.0.0-1762-gcb17f286 (cb17f286272f7ae9dbdf8117ca7b077b0a5cf650) reef (dev)": 1
    },
    "mgr": {
        "ceph version 18.0.0-1762-gcb17f286 (cb17f286272f7ae9dbdf8117ca7b077b0a5cf650) reef (dev)": 2
    },
    "osd": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 8
    },
    "mds": {},
    "rgw": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2
    },
    "overall": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 12,
        "ceph version 18.0.0-1762-gcb17f286 (cb17f286272f7ae9dbdf8117ca7b077b0a5cf650) reef (dev)": 3
    }
}

For whatever reason, later in the test when we try to upgrade just 2 of the OSDs, the "ceph versions" output isn't reporting versions for some of the daemon types. The output comes out like

{
    "mon": {
        "ceph version 18.0.0-1762-gcb17f286 (cb17f286272f7ae9dbdf8117ca7b077b0a5cf650) reef (dev)": 3
    },
    "rgw": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2
    },
    "overall": {
        "ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)": 2,
        "ceph version 18.0.0-1762-gcb17f286 (cb17f286272f7ae9dbdf8117ca7b077b0a5cf650) reef (dev)": 3
    }
}

With no "osd" entry in the output at all, the check fails.

Verified the issue exists in a main branch build with no additional commits

https://pulpito.ceph.com/adking-2023-01-13_05:56:45-orch:cephadm-main-distro-default-smithi/7140915
https://pulpito.ceph.com/adking-2023-01-13_05:56:45-orch:cephadm-main-distro-default-smithi/7140916