Ceph

osd: Faster Peering

Summary

For correctness reasons, peering requires a series of serial message transmissions and filestore syncs prior to completion.  This puts something of a lower bound on the latency client IO suffers on cluster change.

Owners

Sam Just (RedHat)

Interested Parties

Guang Yang (Yahoo!)

Current Status

GetInfo->GetLog->GetMissing requires three round trips to replicas.  First, we get pg infos from every osd in the prior set, acting set, and up set in order to choose an authoritative log.  Second, we fetch the authoritative log.  Last, we fetch missing sets from each acting set replica for use during recovery.

1) Can we preemptively request the log+missing for osds in the most recent prior set interval to hopefully skip the GetLog step?

2) Can we preemptively request the log+missing for acting and up osds in the GetInfo set to hopefully skip the GetMissing step?

Another wrinkle is that replicas do not send the info requested in GetInfo and the primary cannot start peering until the previous acting interval has been flushed.

1) We might be able to relax this to waiting for a commit (journal only) if we track unstable objects across intervals.  We need to track unstable objects for replicas going forward anyway to get replica reads right, so this might not be so bad.