Ceph » rgw

Found this issue using Nautilus and confirmed the issue is still present on Pacific.

After uploading a large object (1.4Tb or more) to a bucket, the bucket delete does not properly cleanup the default.rgw.buckets.data pool.

Repro:
The ceph.conf includes some tweaks to speed up the gc and what i believe will make the multipart uploads more efficient
rgw gc max objs = 2647
rgw gc obj min wait = 300
rgw gc processor max time = 600
rgw gc processor period = 600
rgw multipart min part size = 20971520
rgw multipart part upload limit = 50000

The large object was uploaded to the bucket using the following command:

s5cmd -numworkers 32 --endpoint-url $ENDPOINT_URL --no-verify-ssl cp -c 16 /mnt/upload-nvmedisk/test_data/ s3://$BUCKET_NAME/

The object was then read from the bucket and verified that the upload was successful.

Before deleting the bucket, we can see how many objects are in the rgw.data pool

[root@dr740-44-23 ~]# rados -p default.rgw.buckets.data ls | wc -l
360002

The bucket is then deleted with the following command:

radosgw-admin bucket rm --bucket=$BUCKET_NAME --purge-objects

`radosgw-admin gc list --include-all` shows some objects in the queue to be cleaned up
once they are cleaned up, there are still objects in the rgw.data pool:

[root@qs-42-27 ~]# rados -p default.rgw.buckets.data ls | wc -l
326471

These objects are 'shadow' objects which look like this "2288b7bf-3463-4d1c-8069-4232a61bc3e7.94805.2__shadow_1400G.testdata.sql.2~PD8UdUL-jrH_L6GiB57vi-GCDtNBZRG.3848_28"

The only way I have found to successfully delete these orphaned objects is to delete them all manually with `rados -p default.rgw.buckets.data rm $ORPHAN`