Ceph » RADOS

Description of problem:
WE are observing that the connection scores of the monitor with the highest rank is not populated properly after installation. It is missing the peer information.
More over, we have observed that the peer_reports contain the unexpected rank of -1.

All the mons are part of the quorum and ceph status is healthy..

Tried the scenario on multiple clusters with 3 & 5 mons. In each case mon with highest rank had ranks missing for other peer monitors.

# ceph mon dump
epoch 3
fsid 6b008e40-761f-11ed-9d70-fa163ed6291c
last_changed 2022-12-07T11:15:58.063080+0000
created 2022-12-07T11:08:25.190391+0000
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.0.210.25:3300/0,v1:10.0.210.25:6789/0] mon.ceph-pdhiran-6nkq6t-node1-installer
1: [v2:10.0.209.151:3300/0,v1:10.0.209.151:6789/0] mon.ceph-pdhiran-6nkq6t-node6
2: [v2:10.0.208.155:3300/0,v1:10.0.208.155:6789/0] mon.ceph-pdhiran-6nkq6t-node2
dumped monmap epoch 3

# ceph daemon mon.ceph-pdhiran-6nkq6t-node1-installer  connection scores dump
{
    "rank": 0,
    "epoch": 14,
    "version": 223,
    "half_life": 43200,
    "persist_interval": 10,
    "reports": {
        "report": {
            "rank": -1,
            "epoch": 0,
            "version": 0,
            "peer_scores": {}
        },
        "report": {
            "rank": 0,
            "epoch": 14,
            "version": 223,
            "peer_scores": {
                "peer": {
                    "peer_rank": 1,
                    "peer_score": 0.99995375391737207,
                    "peer_alive": true
                },
                "peer": {
                    "peer_rank": 2,
                    "peer_score": 0.99995374350154376,
                    "peer_alive": true
                }
            }
        },
        "report": {
            "rank": 1,
            "epoch": 14,
            "version": 168,
            "peer_scores": {
                "peer": {
                    "peer_rank": 0,
                    "peer_score": 1,
                    "peer_alive": true
                },
                "peer": {
                    "peer_rank": 2,
                    "peer_score": 0.99736158141292575,
                    "peer_alive": false
                }
            }
        },
        "report": {
            "rank": 2,
            "epoch": 14,
            "version": 111,
            "peer_scores": {
                "peer": {
                    "peer_rank": 0,
                    "peer_score": 1,
                    "peer_alive": true
                }
            }
        }
    }
}

Steps to Reproduce:
1. Bootstrap Ceph cluster
2. Deploy 5 Monitors
3. Observe that connection scores are not populated properly.