Passive monitors


Allow 'passive' monitors, opposed to the currently supported monitors, which are always active participants in the quorum. These 'passive' monitors would be notified and receivers of proposals, but would not be accounted for quorum or proposal commit purposes, nor would they handle client requests. This would allow these monitors to be flipped back into quorum (e.g., in case of a failure of an active monitor) in a fraction of the time it would take to sync a monitor from scratch, or simply act as backups.

Joao Eduardo Luis (SUSE)

Interested Parties
If you are interested in contributing to this blueprint, or want to be a "speaker" during the Summit session, list your name here.
Joao Eduardo Luis (SUSE)
Danny Al-Gaaf (Deutsche Telekom)

Current Status
There is currently no related work. The whole thing will need to be done from scratch, although adding the 'active/passive monitor' concept is theoretically trivial (by adding new fields to the monmap), quite a bit of work will need to be done on Paxos to make sure we send out each proposal to the passive monitors, as well as on the monitor probing, as we will need to know that a given passive monitor is alive and well and ready to receive the proposals as they come. We may need to add a monitor heartbeat if we find that just Paxos timeouts are unable to handle that though, but I don't think that is going to be necessary from the get go.

Detailed Description
We are currently quite inflexible when it comes to monitors and their roles.
Every single monitor in the monmap is accounted for the purpose of forming a quorum, and will be assigned a role of either a peon or a leader upon an election (assuming said monitor is up and running, and reachable by the other monitors in the cluster).
While this is what would be expected on a distributed consensus system, given we need a strict majority of monitors to reliably propose a new value, it does leave us some room for improvement.
What we now propose is having a 'passive' role, to be set on the monmap, which will allow a monitor to passively be updated just the same as any other monitor in the quorum, but without the burden of being an active participant in the quorum.
This should help with maintaining redundancy without the hard requirement of having the monitor being an active participant in the cluster, handling client requests or requiring its acknowledgement when committing a new value.
Not being a go-to monitor by clients will, in a large enough cluster, relieve the monitor from being on a well-performing machine; the monitor may simply act as a rolling backup monitor for disaster recovery, for instance, or to be flipped to active in case of failure on one of the active monitors -- with the added benefit of virtually no synchronization due to this monitor being a receiver of proposals as they are committed in the cluster.

Work items

Coding tasks
Paxos: support out-of-quorum monitors while not accounting them for commit phase
Paxos: correctly handle old acks to committed proposals from passive monitors, sending them the next proposal (from a historical point-of-view)
Monitor: make sure we can figure out whether a monitor is okay to join the quorum as is, or if it needs a sync
Monitor: add means to make sure a store is consistent with the rest of the quorum (because we are likely relaxing consistency to achieve this)

Documentation tasks
How to create/add a passive monitor
Maybe even a blog post about active vs passive monitors