mon: robust bootstrap
Currently mkfs looks like:
- create initial states on each monitor independently
- start them up and they'll form a quorum and go from there.
Instead, we want to
- seed each mon with the mon. key and some initial keys
- on startup, look for an existing quorum. if it exists, sync committed stuff and then join.
- if quorum does not exist, try to form a new one.
- once a quorum is formed, if there is no initial state, create it using the seed data.
Or something along those lines. I suspect we need to take a hard look at how the election stuff currently works so that new monitors can discover each other without disrupting an existing active quorum, and form a new quorom with a subset of the live nodes. Currently the assumption is that alive => will join quorum, which isn't quite what we want.
This will enable a robust parallel mkfs/bootstrap driven by chef/whatever without any ordering requirements.