Ceph

last_epoch_started is used to bound how far back in time we query other OSDs in order to recovery PG state (this is the prior_set). Currently we update last_epoch_started in PG::activate() and broadcast to replicas, but we do this before any state we recovered hits disk. This means we might hear about a last_epoch_started X even though the nodes in X (primary OR replicas) crashed before committing that recovered state to disk.

Instead, we should activate all peers, everyone commit, send acks back to the primary saying "yes, I have committed the recovered pg info and log for this interval", and only then, once all replicas have done so, update last_epoch_started, rebroadcast to replicas, and queue for disk.