Ceph » rgw

Casey Bodley wrote:

how big was this object?

are you familiar with the garbage collection system that cleans up these deleted tail objects? see https://docs.ceph.com/en/latest/radosgw/config-ref/#garbage-collection-settings

you can use `radosgw-admin gc list --include-all` to show all files that are scheduled for garbage collection. by default, garbage collection waits 2 hours (rgw_gc_obj_min_wait) before it cleans up these objects, because clients may have started reading them just before deletion

note that there is a gc bug when objects get too large (like over 1TB) that we're tracking at https://tracker.ceph.com/issues/49823

Thanks, @Casey Bodley! File ~ 300mb ."radosgw-admin --bucket = support-files gc process --include-all" actually cleared the pool. So this is not a bug.

Found another problem. I was creating a test pool and bucket that I already deleted along with the objects - bucket and pool created from dasboard, objects deleted from rados -p rm. Bucket deleted from dashboard, pool from gui pve

But
"radosgw-admin gc list --include-all" shows this

 {
                "pool": "test.bucket.data",
                "oid": "0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2~nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48",
                "key": "",
                "instance": "" 
            }

radosgw-admin process --include-all --debug-ms = 1 does not output errors, but rerun gc list does not show changes
The collector sees non-existent objects. radosgw-admin bucket check --fix did not affect the result
Since the objects in the default pool were manually deleted many times, I created the test.bucket.data pool again and then ran the gc process -> gc list -clean!
It is not obvious behavior that the collector tries to work with non-existent objects and pools and does not give an error message.

Pritha Srivastava wrote:

Can you give me the exact steps that you tried for the second bug that you are reporting,

I understand that you have a test pool, and then you created and objects. And then you deleted them, and when you run a gc list, it gives some o/p, but when you run gc process, it doesn't delete anything? and you still see the same o/p from gc list command? And I understand that these are multipart objects. Is there anything else that you would like to add to this?

1.Created pool test.bucket.data
2.Created a bucket in this pool
3.Added objects to this bucket
4.Deleted objects (First through the browser and then rados -p rm)
5.Removed the bucket
6.Removed pool test.bucket.data

7. "radosgw-admin gc list --include-all" shows this

{
                "pool": "test.bucket.data",
                "oid": "0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2 ~ nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48",
                "key": "",
                "instance": "" 
            }
8. "radosgw-admin gc process --include-all" and "radosgw-admin bucket check --fix" -No error messages
9. Again I do "radosgw-admin gc list --include-all" and again I see this
{
                "pool": "test.bucket.data",
                "oid": "0c4323f1-afd6-44b7-b2e3-e566ed2ba18f.4472377.1__multipart_debian-11.0.0-amd64-netinst.iso.2 ~ nNqExIsOUW-HTmP5uxFqEllTb5OCD0c.48",
                "key": "",
                "instance": "" 
            }
10.radosgw-admin process --include-all --debug-ms = 1 does not output errors
11.radosgw-admin gc list --include-all everything is still
12.Create pool test.bucket.data
13.radosgw-admin process --include-all
14.radosgw-admin gc list --include-all output is now clean.