RGW misplaces index entries after dynamically resharding bucket
When RGW reshards buckets with ~250k index entries*, I've noticed some s3:PutObject requests that return 200 end up with index entries under the old index shard oids, presumably recreated after dynamically resharding deleted the original shards. These objects can be retrieved via s3:GetObject, since that doesn't typically look for an index entry, but s3:ListObjects doesn't show any information about the successfully written object.
This does not happen reliably. I've been able to recreate this in a staging environment in 1-4 of every 30 buckets, with and without rgw caching enabled on the rgw nodes serving the s3:PutObject requests. mtime on the misplaced index entries shows a burst of writes (all within a few seconds) going to the wrong index after resharding completes, with the majority of writes going to the correct index. This burst can happen multiple times after resharding, from 20s after resharding completes to 12m. I haven't spotted any useful error debug logging in the client logs, but have also not turned up logging very high at all, in part due the volume of requests required for recreating this issue. Let me know if there's any debug information that would be useful. Thanks for reading!
- I don't think 250k is significant in any way, it's just the threshold at which we tend to reshard the most buckets.