Ceph » rgw

When a S3 PUT operation creates a new object in a versioned bucket, the bucket index instance entry is created and bucket stats are incremented prior to the object being "linked" into the index via the creation of an associated plain-versioned index entry. When a S3 DELETE operation deletes an object from a versioned bucket, the process happens in reverse; the object is "unlinked" from the index, via the deletion of the associated plain-versioned index entry, prior to the instance entry being removed and the bucket stats being decremented. The instance entries are therefore the "source of truth" for bucket stats with respect to these operations.

When a bucket is resharded, the bucket stats for each shard must be recomputed since each shard stores its own stats in the header. The current code that performs the accounting for stats on new index shards during the resharding process uses only plain index entries. Likewise, the code that implements the `radosgw-admin bucket check --fix` operation also uses only plain index entries. The plain entries are therefore the "source of truth" for bucket stats with respect to these operations.

This dichotomy between how the stats are derived permits inaccuracies. For example, consider the case where a PUT operation is already in progress when a reshard starts on the relevant bucket. The reshard starts after the PUT operation had created the object and accounted for the new object version in the "old" bucket header stats, but hadn't yet "linked" the object into the index by creating the associated plain entry. Because the plain-versioned entry is not yet created, the reshard would not include this object in the new bucket shard's stats - the stats now show an object count and bucket size that are less than the true values. Let's say the reshard completes, the PUT operation finishes, and we subsequently delete all objects in the bucket. The bucket stats will roll under zero because the DELETE operation will decrement the size of our subject object despite the fact that it wasn't accounted for in the stats. We end up with bucket stats reporting usage just under 2^64. The same scenario can happen with a DELETE request that is in progress during a reshard or with other "unlinked" objects that can be created as a result of errors that interrupt bucket index transactions.

To solve the issue, we should consistently use instance entries as the "source of truth" for bucket stat accounting, across all relevant operations.