Ceph » rgw

The test_versioning_obj_create_versions_remove_special_names test causes leftover objects in the RGW data pool. This exists at least as far back as Hammer in my testing; and still exists at least as of jewel commit 7a94d9767b3dde5b9cdde2a438ba288944a1a7a6.

Reproduction:
1. Take a clean RGW (data pool empty of objects or knowing exactly what is present)
2. Run the test_versioning_obj_create_versions_remove_special_names test
3. Ensure the bucket was removed (deleting everything via list_keys, list_versions and list_multiparts first).
3. Look at leftover objects in the data pool

Resolutions:
(for jewel)
1. The leak needs to be fixed.
(for kraken)
2. There needs to be a way to fix old instances of any similar leftover objects.

Bucket.instance metadata
{
    "key": "bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945",
    "ver": {
        "tag": "__FLSptEZessAacktjTkfwEW",
        "ver": 6
    },
    "mtime": "2016-04-09 18:14:53.440121Z",
    "data": {
        "bucket_info": {
            "bucket": {
                "name": "robbat2ym5g17p942xg9fbzanbjer-1",
                "pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_pool": "default.rgw.buckets.index",
                "marker": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945",
                "bucket_id": "481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945" 
            },
            "creation_time": "0.000000",
            "owner": "test1",
            "flags": 2,
            "zonegroup": "93792573-f350-4c11-be12-d9fb9e0f8f08",
            "placement_rule": "default-placement",
            "has_instance_obj": "true",
            "quota": {
                "enabled": false,
                "max_size_kb": -1,
                "max_objects": -1
            },
            "num_shards": 0,
            "bi_shard_hash_type": 0,
            "requester_pays": "false",
            "has_website": "false",
            "swift_versioning": "false",
            "swift_ver_location": "" 
        },
        "attrs": [
            {
                "key": "user.rgw.acl",
                "val": "AgJ5AAAAAwISAAAABQAAAHRlc3QxBQAAAHRlc3QxAwNbAAAAAQEAAAAFAAAAdGVzdDEPAAAAAQAAAAUAAAB0ZXN0MQQDMgAAAAICBAAAAAAAAAAFAAAAdGVzdDEAAAAAAAAAAAICBAAAAA8AAAAFAAAAdGVzdDEAAAAAAAAAAA==" 
            },
            {
                "key": "user.rgw.idtag",
                "val": "" 
            },
            {
                "key": "user.rgw.manifest",
                "val": "" 
            }
        ]
    }
}

Leftover objects (listing the contents of each pool via librados python):
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_ ,nspace=,locator=None)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945_:,nspace=,locator=None)
Pool=default.rgw.buckets.data Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945___testobj,nspace=,locator=481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945__testobj)
Pool=default.rgw.buckets.index Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None)
Pool=default.rgw.data.root Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=.bucket.meta.robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945,nspace=,locator=None)
Pool=default.rgw.log Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff058455360>,key=2016-04-09-18-481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945-robbat2ym5g17p942xg9fbzanbjer-1,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:1,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:2,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:3,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:4,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:5,nspace=,locator=None)
Pool=default.rgw.meta Object rados.Object(ioctx=<rados.Ioctx object at 0x7ff0584552f0>,key=.meta:bucket.instance:robbat2ym5g17p942xg9fbzanbjer-1:481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945:__FLSptEZessAacktjTkfwEW:6,nspace=,locator=None)

# rados -p default.rgw.buckets.index listomapkeys .dir.481ec65f-7f93-4892-b0c0-8a367a62719b.40025.945 

:
__
__testobj
1001_ 
1001_:
1001___
1001___testobj
<pre>