Ceph » rgw

I've spent some time trying to properly understand the mechanics of what's going on here. In particular, I began to wonder why deletion is a problem specifically, and updates don't suffer the same re-replication issue. It turns out that the reason is in how write conflicts are resolved -- should A, B both have an update at similar times, then the highest timestamp wins. This works (ignoring clock skew) fine for updates, but we can't check the timestamp at which a delete operation happened -- we get the timestamp from the object, which no longer exists. As a result, later deletes are always overwritten by older puts should a conflict arise.

Now, conflicts such as this are uncommon -- first you need a workload with heavy PUT/DELETE traffic, such that the replication and deletion can race to begin with. Secondly, in the set up described above, the race has to occur during full sync -- incremental sync uses the zone trace to skip the re-replications -- but if both sides were accepting traffic, then such a conflict could occur at any time. Thirdly, there exists a mechanism already in the RGW to resolve the conflicts, but it has limitations, which I'll discuss now.

In the RGWRados class, we have an LRU map (the tombstone cache) of recently deleted objects and the time of the deletes, which is checked during writes. If we find a later timestamp for our object, then we don't bother doing the write, since it's older than the delete. This would solve the issue! However, the LRU map is limited:

1. (unless I'm missing something) it's instance local -- if a cluster has multiple RGWs accepting traffic, then there's no guarantee that we will hit the tombstone cache even if there's been a more recent delete than our write, if it was to a different RGW. In the cluster we found this, we have some RGW instances dedicated to replication only, and not serving client requests. These will never hit the tombstone cache in the event of a conflict!

2. the size of the map is finite. This size is tunable already, but it is possible to overload the map and end up in the sticky situation of non-matching zones.

In short, the issue is that we can't reliably tell when a delete of an object happened, so we will always replicate the object creation, even if the delete is later. There are a couple of ways we could go about resolving this, I think.

1. Use tombstones instead of completely deleting objects
2. Keep a cache of tombstones in RADOS rather than the RGW

Of the two, I think (2) is less work and less of a departure from the current state of affairs, so is probably the best course to investigate first.

I'm happy to work on a fix to this, but don't have the power to assign this to myself!

Casey Bodley wrote:

1) when writing a replicated object, fetch_remote_obj() adds an object attribute RGW_ATTR_OBJ_REPLICATION_TRACE containing the zone trace. this work is partially completed in https://github.com/ceph/ceph/pull/49767 but needs more testing

https://github.com/ceph/ceph/pull/49767 is ready for review and testing

We've opened https://github.com/ceph/ceph/pull/51715 to address this, which should be ready for review and testing. I've hammered these changes with the workload above for several hours without hitting the issue (without the fix, I can reliably reproduce within a few runs).