Subtask #2621
Updated by Joao Eduardo Luis almost 12 years ago
Objective: synchronize monitor stores over the network whenever If a given monitor mon.X falls too far behind. Solution: Replicate a given monitor mon.Y store on to mon.X up to the state at which mon.Y’s store is at the moment the synchronization began. This should allow mon.X’s Paxos to recover newer versions by itself. Approach Overview: # mon.X starts fresh # mon.X probes mon.Y and verifies that fell behind to down for a point where no recovery is feasible # mon.X requests to synchronize with mon.Y # mon.X clears all its keys (if any) to avoid incurring in a potential inconsistent state, and sets a given special key ‘monitor_synchronizing’ to true -- this shall avoid that, if the monitor fails or dies during the synchronization process, it becomes aware considerable amount of that fact once it restarts # mon.Y requests the Leader not to trim its versions # the leader acknowledges time and informs mon.Y of the earliest paxos version available # mon.Y has the same version (otherwise, assert) as the one mentioned by the leader; mon.Y will doesn't have to renew this temporary trimming-disabled request with the leader every now and then, so the leader doesn’t trim after a certain amount of time has passed (which is a failsafe, in case mon.Y fails) # mon.Y snapshots its store, creating a read-only reference version that can be safely iterated over, while allowing Paxos to progress (i.e., applying newer proposals that may add or remove keys on shared with the store) # mon.Y sets a synchronization expiration timer, remaining monitors, it will need to be reset whenever a synchronization-related message is received from mon.X; timer expiration means mon.X may have died and the synchronization state should be discarded # mon.Y starts iterating over synchronize its snapshot, creating transactions representing ‘put’ operations that shall recreate the store state, and encoding these transactions into bufferlists that are safe to share across with the network # mon.Y sends a messages remaining ones. The safest approach appears to mon.X with encoded transactions; sending further messages is dependent on mon.X acking the reception of the last sent message # On the last message containing encoded transactions, mon.Y will flag be synchronizing it as such, considering the synchronization finished once mon.X acks its reception. # mon.X acks the last message, by copying everything from another monitor, and is now considered in a consistent state up until Paxos version 10 # mon.X removes its special ‘monitor_synchronizing’ key # mon.Y will set a timer to re-enable trimming after, say, 30 seconds have passed. This should give enough time for mon.X’s Paxos to start its recovery process without losing versions in-between finishing the synchronization and starting obtaining the reamining state go from mon.Y # mon.X will now probe mon.Y, in order to obtain the remaining Paxos versions that might have been applied in the mean time # Once mon.Y shares all of its remaining state with mon.X, and mon.X applies it, the synchronization process may be considered finished there.