Mon - PaxosServices relying on hooks instead of hardcoded order to updatepropose¶

Summary¶

There are some dependencies on some PaxosServices. For instance, on several occasions the MDSMonitor will need to have the OSDMonitor to issue a proposal of changes to the OSDMap. On the other hand, the PGMonitor relies on the OSDMonitor to have an initialized state to perform some operations during 'update_from_paxos()'.
This imposes a strict ordering when initializing and updating the services states, which is not obvious at all at the moment. With the addition of new features, this becomes a fragile mechanism which can break in subtle ways and typically hard to identify.
We intend to make the monitor more robust by clearly defining the dependencies between services, and relying on hooks instead of a strict ordering enforced by the Monitor class.

Owners¶

• Joao Eduardo Luis (Inktank)
• Name (Affiliation)
• Name

Interested Parties¶

• Name (Affiliation)
• Name (Affiliation)
• Name

Current Status¶

Summary says it all.

Detailed Description¶

One recent issue forced us to patch the Monitor class in such a way that we have the MonmapMonitor being alerted to the fact that an election has finished before any other PaxosService, so that we guarantee that the very first proposal in the cluster is always a monmap with all the participants in the very first election. Enforcing this sort of requirements tend to pollute the code and make it hard to build upon the existing code.

A possibly realistic approach to avoid having to explicitly order every call to every service could be relying on hooks.

For instance, say that we want to keep the new behavior introduced by the linked patch above. We could have the Monitor exposing a 'register_on_election_finished()' function, which would be called by any and all PaxosServices that actually wanted to know about a finished election. We could have the MonmapMonitor register to it. Then we could have every PaxosService exposing a similar (if not the same) interface, and have all the other PaxosServices registering on MonmapMonitor's. This would almost impose the same string ordering, if not for the fact that we actually enforce a string ordering of PaxosServices on the 'paxos_service' vector that is being iterated on the above patch -- having all the services relying on MonmapMonitor's election_finished() hook, without enforcing a predetermined order could be chaotic, or it could work just like a charm: there's no way to tell which at the moment, as there is no information regarding who depends on what. Nevertheless, say that the PGMonitor would require to run before the OSDMonitor's; we could have the PGMonitor registering on a 'register_pre_on_election_finished()' from the OSDMonitor class, that would be called before the OSDMonitor's hook on_election_finished().

This approach hasn't been fully thought through, but once the requirements and dependencies are established and documented, having each service declaring their dependencies should be more obvious and readable in the long-term than having them enforced by the Monitor class in a manner that is neither, nor robust enough to tolerate change.

Work items¶

Coding tasks¶

1. mon: assess PaxosService's dependencies on state changes of other PaxosServices
2. mon: define a proper hook interface on a per-PaxosService basis or a monitor-wide basis (or some other basis I guess)
3. mon: actually fix this for the good of us all

Build / release tasks¶

1. Task 1
2. Task 2
3. Task 3

Documentation tasks¶

1. mon: clearly document all the dependencies between PaxosServices.
   • Even if this blueprint doesn't go ahead, lets have this one done.