Project

General

Profile

Bug #29419

Auto deleting is not working with multitenancy enabled

Added by Jerzy Midura 4 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Target version:
Start date:
08/28/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

The issue was found on Jewel 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) and confirmed also in Luminous 12.2.5 (cad919881333ac92274171586c827e01f554a70a).

When multitenancy for rgw is enabled in ceph.conf file:

rgw keystone implicit tenants = true

Auto delete function is not working properly.

Test procedure as follows.

Check if there are objects in the pool and available containers:

[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
[root@luminous-osc ~(keystone_admin)]# radosgw-admin bucket list
[
    "tempest-TestContainer-1849671326",
    "tempest-TestContainer-1305170331" 
]


There are only some leftovers made by tempest.

Create some file and upload it to Swift with delete-after header:

[root@luminous-osc ~(keystone_admin)]# uptime > autodeletefile
[root@luminous-osc ~(keystone_admin)]# swift upload autodeleteContainer autodeletefile --header "X-Delete-After: 30" 

Check if file is available via swift:

[root@luminous-osc ~(keystone_admin)]# swift stat autodeleteContainer autodeletefile
               Account: v1
             Container: autodeleteContainer
                Object: autodeletefile
          Content Type: binary/octet-stream
        Content Length: 71
         Last Modified: Tue, 28 Aug 2018 08:31:37 GMT
                  ETag: c721f06d83ae2e2be3fa104ba7e79d13
            Meta Mtime: 1535445040.199022
           X-Delete-At: 1535445127
         Accept-Ranges: bytes
           X-Timestamp: 1535445097.44745
            X-Trans-Id: tx00000000000000002f070-005b85087d-7e420-default
X-Openstack-Request-Id: tx00000000000000002f070-005b85087d-7e420-default

Check again after 30 seconds:

[root@luminous-osc ~(keystone_admin)]# swift stat autodeleteContainer autodeletefile
Object HEAD failed: http://106.120.114.157:8080/swift/v1/autodeleteContainer/autodeletefile 404 Not Found
Failed Transaction ID: tx00000000000000002f20b-005b850917-7e420-default

But the object is still present in the pool:

[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
342db702-df7e-4760-bacd-68341e78ede6.517152.3_autodeletefile
[root@luminous-osc ~(keystone_admin)]# radosgw-admin bucket list
[
    "tempest-TestContainer-1849671326",
    "tempest-TestContainer-1305170331",
    "0eb72f67c1c74beca6f84432f8cc0ee9/autodeleteContainer" 
]

Run the cleanup manually:

[root@luminous-osc ~(keystone_admin)]# radosgw-admin objects expire --debug-rgw=20
2018-08-28 17:37:29.827911 7f54d1c4b700  2 RGWDataChangesLog::ChangesRenewThread: start
...
2018-08-28 17:37:30.117161 7f54ecbeec40 15 got removal hint for: 1535445127 - 0eb72f67c1c74beca6f84432f8cc0ee9:autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3:autodeletefile:
2018-08-28 17:37:30.117222 7f54ecbeec40 20 reading from default.rgw.meta:root:.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3
2018-08-28 17:37:30.117237 7f54ecbeec40 20 get_system_obj_state: rctx=0x7ffcc14f53d0 obj=default.rgw.meta:root:.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 state=0x55d090ca7bb0 s->prefetch_data=0
2018-08-28 17:37:30.117243 7f54ecbeec40 10 cache get: name=default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 : miss
2018-08-28 17:37:30.118078 7f54ecbeec40 10 cache put: name=default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 info.flags=0x0
2018-08-28 17:37:30.118114 7f54ecbeec40 10 adding default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 to cache LRU end
2018-08-28 17:37:30.118121 7f54ecbeec40 15 NOTICE: cannot find bucket = autodeleteContainer. The object must be already removed
2018-08-28 17:37:30.118124 7f54ecbeec40 15 not actual hint for object: autodeletefile
2018-08-28 17:37:30.118135 7f54ecbeec40 20 trying to trim removal hints to=2018-08-28 17:37:29.887783, to_marker=1_1535445127.444034_0eb72f67c1c74beca6f84432f8cc0ee9:autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3:autodeletefile:
...

Object is still present:

[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
342db702-df7e-4760-bacd-68341e78ede6.517152.3_autodeletefile

Object can be deleted via swift api but with the following error:

[root@luminous-osc ~(keystone_admin)]# swift delete autodeleteContainer autodeletefile
Error Deleting: autodeleteContainer/autodeletefile: Object DELETE failed: http://106.120.114.157:8080/swift/v1/autodeleteContainer/autodeletefile 404 Not Found   NoSuchKey
[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745

History

#1 Updated by Jerzy Midura 4 months ago

Important notice: on the same environment without

rgw keystone implicit tenants = true

The autodelete function is working fine.

#2 Updated by John Spray 3 months ago

  • Project changed from Ceph to rgw

#3 Updated by Abhishek Lekshmanan 3 months ago

  • Assignee set to Abhishek Lekshmanan

Also available in: Atom PDF