Bug #15445
multisite s3tests test_versioning_obj_create_versions_remove_special_names causes rados object leaks
0%
Description
The test_versioning_obj_create_versions_remove_special_names test causes leftover objects in the RGW data pool. This exists at least as far back as Hammer in my testing; and still exists at least as of jewel commit 7a94d9767b3dde5b9cdde2a438ba288944a1a7a6.
Reproduction:
1. Take a clean RGW (data pool empty of objects or knowing exactly what is present)
2. Run the test_versioning_obj_create_versions_remove_special_names test
3. Ensure the bucket was removed (deleting everything via list_keys, list_versions and list_multiparts first).
3. Look at leftover objects in the data pool
Resolutions:
(for jewel)
1. The leak needs to be fixed.
(for kraken)
2. There needs to be a way to fix old instances of any similar leftover objects.
Bucket.instance metadata { "key": "bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945", "ver": { "tag": "__FLSptEZessAacktjTkfwEW", "ver": 6 }, "mtime": "2016-04-09 18:14:53.440121Z", "data": { "bucket_info": { "bucket": { "name": "robbat2ym5g17p942xg9fbzanbjer-1", "pool": "default.rgw.buckets.data", "data_extra_pool": "default.rgw.buckets.non-ec", "index_pool": "default.rgw.buckets.index", "marker": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945", "bucket_id": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945" }, "creation_time": "0.000000", "owner": "test1", "flags": 2, "zonegroup": "93792573-f350-4c11-be12-d9fb9e0f8f08", "placement_rule": "default-placement", "has_instance_obj": "true", "quota": { "enabled": false, "max_size_kb": -1, "max_objects": -1 }, "num_shards": 0, "bi_shard_hash_type": 0, "requester_pays": "false", "has_website": "false", "swift_versioning": "false", "swift_ver_location": "" }, "attrs": [ { "key": "user.rgw.acl", "val": "AgJ5AAAAAwISAAAABQAAAHRlc3QxBQAAAHRlc3QxAwNbAAAAAQEAAAAFAAAAdGVzdDEPAAAAAQAAAAUAAAB0ZXN0MQQDMgAAAAICBAAAAAAAAAAFAAAAdGVzdDEAAAAAAAAAAAICBAAAAA8AAAAFAAAAdGVzdDEAAAAAAAAAAA==" }, { "key": "user.rgw.idtag", "val": "" }, { "key": "user.rgw.manifest", "val": "" } ] } } Leftover objects (listing the contents of each pool via librados python): Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__) Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_ ,nspace=,locator=None) Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_:,nspace=,locator=None) Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___testobj,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__testobj) Pool=default.rgw.buckets.index Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None) Pool=default.rgw.data.root Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.bucket.meta.robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None) Pool=default.rgw.log Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=2016-04-09-18-481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945-robbat2ym5g17p942xg9fbzanbjer-1,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:1,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:2,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:3,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:4,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:5,nspace=,locator=None) Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:6,nspace=,locator=None) # rados -p default.rgw.buckets.index listomapkeys .dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945 : __ __testobj 1001_ 1001_: 1001___ 1001___testobj <pre>
History
#1 Updated by Orit Wasserman almost 8 years ago
- Status changed from New to Duplicate
duplicate of #15318
#2 Updated by Robin Johnson almost 8 years ago
@orit:
This is not a duplicate; Note that I said this was as of commit 7a94d9767b3dde5b9cdde2a438ba288944a1a7a6, which DOES already include your fix from PR#8488.
#3 Updated by Orit Wasserman almost 8 years ago
- Status changed from Duplicate to New
#4 Updated by Orit Wasserman almost 8 years ago
- Assignee set to Orit Wasserman
#5 Updated by Sage Weil almost 8 years ago
- Priority changed from High to Urgent
#6 Updated by Orit Wasserman almost 8 years ago
#7 Updated by Orit Wasserman almost 8 years ago
- Priority changed from Urgent to High
In multisite configuration we cannot remove the bucket instance because of the asynchronous replication.
We need to create a special tool to remove the leaked bucket instances in such scenarios.
It will need to check that the synchronization is done for this bucket and only than remove the bucket instance.
#8 Updated by Orit Wasserman almost 8 years ago
- Subject changed from s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks to multisite s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks
- Status changed from New to 12
#9 Updated by Orit Wasserman almost 8 years ago
- Assignee changed from Orit Wasserman to Casey Bodley
In the new multisite it will be part of the log trimmer
#10 Updated by Loïc Dachary over 7 years ago
- Backport changed from hammer, infernalis to hammer
infernalis is EOL
#11 Updated by Yehuda Sadeh over 7 years ago
- Assignee changed from Casey Bodley to Orit Wasserman
Orit, can you try reproducing it? It seems to me that there's an issue with certain rgw objects that aren't getting removed correctly there, but it's hard to be certain. Not sure it's related to the log trimming issue.
#12 Updated by Orit Wasserman over 7 years ago
- Assignee changed from Orit Wasserman to Casey Bodley
We cannot remove the bucket instance on a syncing zone.
Log trimming should fix it.
#13 Updated by Orit Wasserman over 7 years ago
- Backport deleted (
hammer)
#14 Updated by Yehuda Sadeh over 7 years ago
- Subject changed from multisite s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks to multisite s3tests test_versioning_obj_create_versions_remove_special_names causes rados object leaks
#15 Updated by Orit Wasserman over 6 years ago
- Status changed from 12 to Won't Fix
- Affected Versions v0.94.9 added
Won't be fixed in hammer