Ceph » rgw

I realise that the title may be misleading! To clarify, the issue is that deleted object versions that were removed on the primary side are still present on the secondary side, not that replication of deletes directly sent to the secondary side aren't working.

Tom Coldrick wrote:

At the moment I'm not sure on the mechanism by which the DELETE is missed during replication.

I've now done a bunch more debugging, and I think I have a good grasp on what's going on now! First some preliminaries: in the setup used we have two zones (let's call them P and S for primary and secondary) which are using active-active sync. All client traffic is going to the P zone. The issue occurs when the initial full sync from P to S for the bucket fails and needs to be rerun, while traffic is still hitting P. What happens for one of the extra objects is this:

1. Object instance is placed in P by client traffic, entered into bilog with marker M
2. Full sync begins P->S, markers are set for incremental sync
3. Object instance is replicated P->S
4. Object instance is deleted in P, entered into bilog with marker M+t1
5. Some error happens in replication process on S, halting progress
6. Full sync restarts P->S, markers are set again for incremental sync to some M+t2 > M+t1
7. Full sync completes P->S, incremental sync starts from marker M+t2

This means that the delete is completely missed from the bucket sync, and we get permanently out of sync with the P zone.

The title of this issue specifies that the issue is on versioned buckets, but I don't think it would be restricted to them! I can only reproduce this on versioned buckets, but this is because they seem to suffer from EIO errors when attempting to link an OLH sometimes. I've not yet hunted down why this happens, but it was these @EIO@s that were causing the full sync to fail and subsequently re-update the initial positions for the incremental markers.

I gathered a little more evidence for this by modifying InitBucketFullSyncStatusCR to only update the incremental sync shards if the state is in Init, and only to reset the state to Init if the current state is incremental, preventing markers being updated on the retry. This means that the markers used are those from the initial full sync run, and the bilog entries don't get missed. The results are that I can no longer reproduce this issue, but I now hit #58911 on every run. I don't think this is the correct fix in any case -- I just wanted some evidence for the hypothesis.

I have noticed that some bilog shards don't seem to be syncing incrementally, but I haven't yet followed this track in depth.

This turned out to be inconsequential -- some bucket shards simply did not require incremental sync after full sync completed, and so were not syncing incrementally.