Project

General

Profile

Bug #15445

multisite s3tests test_versioning_obj_create_versions_remove_special_names causes rados object leaks

Added by Robin Johnson almost 8 years ago. Updated over 6 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
other
Tags:
rgw, leak
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The test_versioning_obj_create_versions_remove_special_names test causes leftover objects in the RGW data pool. This exists at least as far back as Hammer in my testing; and still exists at least as of jewel commit 7a94d9767b3dde5b9cdde2a438ba288944a1a7a6.

Reproduction:
1. Take a clean RGW (data pool empty of objects or knowing exactly what is present)
2. Run the test_versioning_obj_create_versions_remove_special_names test
3. Ensure the bucket was removed (deleting everything via list_keys, list_versions and list_multiparts first).
3. Look at leftover objects in the data pool

Resolutions:
(for jewel)
1. The leak needs to be fixed.
(for kraken)
2. There needs to be a way to fix old instances of any similar leftover objects.

Bucket.instance metadata
{
    "key": "bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945",
    "ver": {
        "tag": "__FLSptEZessAacktjTkfwEW",
        "ver": 6
    },
    "mtime": "2016-04-09 18:14:53.440121Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "robbat2ym5g17p942xg9fbzanbjer-1",
                "pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_pool": "default.rgw.buckets.index",
                "marker": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945",
                "bucket_id": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945" 
            },
            "creation_time": "0.000000",
            "owner": "test1",
            "flags": 2,
            "zonegroup": "93792573-f350-4c11-be12-d9fb9e0f8f08",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "max_size_kb": -1,
                "max_objects": -1
            },
            "num_shards": 0,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgJ5AAAAAwISAAAABQAAAHRlc3QxBQAAAHRlc3QxAwNbAAAAAQEAAAAFAAAAdGVzdDEPAAAAAQAAAAUAAAB0ZXN0MQQDMgAAAAICBAAAAAAAAAAFAAAAdGVzdDEAAAAAAAAAAAICBAAAAA8AAAAFAAAAdGVzdDEAAAAAAAAAAA==" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            },
            {
                "key": "user.rgw.manifest",
                "val": "" 
            }
        ]
    }
}

Leftover objects (listing the contents of each pool via librados python):
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_ ,nspace=,locator=None)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_:,nspace=,locator=None)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___testobj,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__testobj)
Pool=default.rgw.buckets.index Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None)
Pool=default.rgw.data.root Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.bucket.meta.robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None)
Pool=default.rgw.log Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=2016-04-09-18-481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945-robbat2ym5g17p942xg9fbzanbjer-1,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:1,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:2,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:3,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:4,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:5,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:6,nspace=,locator=None)

# rados -p default.rgw.buckets.index listomapkeys .dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945 

:
__
__testobj
1001_ 
1001_:
1001___
1001___testobj
<pre>

History

#1 Updated by Orit Wasserman almost 8 years ago

  • Status changed from New to Duplicate

duplicate of #15318

#2 Updated by Robin Johnson almost 8 years ago

@orit:
This is not a duplicate; Note that I said this was as of commit 7a94d9767b3dde5b9cdde2a438ba288944a1a7a6, which DOES already include your fix from PR#8488.

#3 Updated by Orit Wasserman almost 8 years ago

  • Status changed from Duplicate to New

#4 Updated by Orit Wasserman almost 8 years ago

  • Assignee set to Orit Wasserman

#5 Updated by Sage Weil almost 8 years ago

  • Priority changed from High to Urgent

#7 Updated by Orit Wasserman almost 8 years ago

  • Priority changed from Urgent to High

In multisite configuration we cannot remove the bucket instance because of the asynchronous replication.
We need to create a special tool to remove the leaked bucket instances in such scenarios.
It will need to check that the synchronization is done for this bucket and only than remove the bucket instance.

#8 Updated by Orit Wasserman almost 8 years ago

  • Subject changed from s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks to multisite s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks
  • Status changed from New to 12

#9 Updated by Orit Wasserman almost 8 years ago

  • Assignee changed from Orit Wasserman to Casey Bodley

In the new multisite it will be part of the log trimmer

#10 Updated by Loïc Dachary over 7 years ago

  • Backport changed from hammer, infernalis to hammer

infernalis is EOL

#11 Updated by Yehuda Sadeh over 7 years ago

  • Assignee changed from Casey Bodley to Orit Wasserman

Orit, can you try reproducing it? It seems to me that there's an issue with certain rgw objects that aren't getting removed correctly there, but it's hard to be certain. Not sure it's related to the log trimming issue.

#12 Updated by Orit Wasserman over 7 years ago

  • Assignee changed from Orit Wasserman to Casey Bodley

We cannot remove the bucket instance on a syncing zone.
Log trimming should fix it.

#13 Updated by Orit Wasserman over 7 years ago

  • Backport deleted (hammer)

#14 Updated by Yehuda Sadeh over 7 years ago

  • Subject changed from multisite s3tests test_versioning_obj_create_versions_remove_special_names causes RGW object leaks to multisite s3tests test_versioning_obj_create_versions_remove_special_names causes rados object leaks

#15 Updated by Orit Wasserman over 6 years ago

  • Status changed from 12 to Won't Fix
  • Affected Versions v0.94.9 added

Won't be fixed in hammer

Also available in: Atom PDF