Eviction still raced with scrub due to preemption
We put code in cache tier eviction to check the scrub range, but that isn't sufficient. During scrub preemption restart we are selecting a smaller chunk. This allows eviction to evict something that had already been sent to the replica. The cleaned_meta_map can buffer the ScrubMap for a new evicted snapshot. The eviction will be received by the replica and the head object will be modified before _scan_snaps() is called with the removed snapshot still present.
snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired
Piecing together the sequence of events:
Chunk including smithi16216580-301...:2e but NOT its head object sent to replica
Replica stores smithi16216580-301...:2e in cleaned_meta_map
Preemption causes new chunk which does NOT include smithi16216580-301...:2e
Eviction is able to delete smithi16216580-301...:2e from cache tier
Replica sees delete and removes snapshot from head object
Scrub gets back to handling a chunk which includes smithi16216580-301...:head
Replica _scan_snaps() outputs error because snapshot 2e is present but head doesn't include it.