Feature #62552
openrgw: provide rgw-orphan-list a way to work with indexless buckets
0%
Description
This feature request originates from Madhavi Kasturi.
Currently rgw-orphan-list will error out if it encounters an indexless bucket. The reason is that it works by comparing the actual rados data objects against those expected by inspecting the indexes. Anything in the former group that's not in the latter group is considered an orphan, But objects in indexless buckets meet those criteria.
I believe there's a better algorithm. We can look for head objects that have the markers of indexless buckets and we can examine their manifests to determine their tail objects (including those that use a different marker due to a shallow copy). Those objects cannot be listed as orphans, so should be subtracted out from any list of orphans produced.
Updated by Casey Bodley 9 months ago
thanks Eric,
if i remember the current algorithm, we're starting with the raw rados ls output, then filtering that based on the output of radosgw-admin bucket radoslist commands to identity the rest as orphans?
this handling of indexless buckets could be a later filtering step, so we only have to look up the bucket metadata (to check for indexless) for entries that we'd otherwise consider as orphans
but given just the rados object names starting with a bucket marker string, how do we reliably look up their bucket metadata? for one, the bucket marker may not match the current bucket instance id. and then, the bucket instance metadata is identified by "bucket-name:instance-id" and we don't know the bucket names
i suppose we'd have to get a listing of all bucket instance metadata from radosgw-admin metadata list bucket.instance and grep for the instance-id, then load that bucket instance metadata to check whether its marker matches
Updated by J. Eric Ivancich 8 months ago
- Tags set to low-hanging-fruit
I think this might make a good summer project (e,g., Google Summer of Code, Outreachy) or a nice starter project for someone. I've added the low-hanging-fruit tag.