Ceph » RADOS

The structure in question is the mon_status output, so it would be useful if you could look at the output of the mon_status command on a mon.

ceph mon_status -f json | jq '.outside_quorum'
[]

^^ HEALTH_OK

ceph mon_status -f json | jq '.outside_quorum'
[]

^^ [WRN] overall HEALTH_WARN 1/3 mons down

So, the problem is that "outside_quorum" never gets set.

This is on a Ceph 12.2.8 (test) setup

@John: your turn ;-)

`outside quorum` does not pertain to down monitors. We may change that if people think it's more obvious, but the main purpose of this structure is to help understanding which monitors have been responding to probes for election purposes. After a successful election, `outside_quorum` is cleared.

Let me see if I get this right.

'After a successful election, `outside_quorum` is cleared."

^^ Do I understand correctly that "outside_quorum" gets cleared, even if there's is one monitor down (or more), when the remaining monitors have held a succesful election?

If that's true, than yes, at least to me that is not obvious. I use that metric in a Ceph dashboard to indicate a mon outide quorum. If there would be another metric, i.e. mon_down, that would also be o.k.

Yes, `outside quorum` is solely used to track which monitors are outside of the quorum during an election; once the election is over, that information is no longer relevant (for the intent it has been originally tracked).

We also have no other means of tracking which monitors are down: if they are not in quorum, they are presumed "down" (although we really have no idea, at the moment, whether they are dead, or unreachable, or not participating in the election).

Follow up conversation on IRC has shown that more information being exposed, or (at least) changing the semantics of what is exposed in `outside quorum` would be useful.

We could add a new field for monitors that are... not part of the quorum, but I'm not sure what I'd call it if not "outside_quorum".

@Sage Weil: indeed :-).

Maybe rename the original use for "outside_quorum" to "outside_election" or something similar to indicate mons that do not play well with the rest of the monitors? And make "outside_quorum" a list of mon(s) that are not part of the quorum (but according to cluster config should be in there).