Bug #29419
Auto deleting is not working with multitenancy enabled
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The issue was found on Jewel 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) and confirmed also in Luminous 12.2.5 (cad919881333ac92274171586c827e01f554a70a).
When multitenancy for rgw is enabled in ceph.conf file:
rgw keystone implicit tenants = true
Auto delete function is not working properly.
Test procedure as follows.
Check if there are objects in the pool and available containers:
[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
[root@luminous-osc ~(keystone_admin)]# radosgw-admin bucket list
[
"tempest-TestContainer-1849671326",
"tempest-TestContainer-1305170331"
]
There are only some leftovers made by tempest.
Create some file and upload it to Swift with delete-after header:
[root@luminous-osc ~(keystone_admin)]# uptime > autodeletefile
[root@luminous-osc ~(keystone_admin)]# swift upload autodeleteContainer autodeletefile --header "X-Delete-After: 30"
Check if file is available via swift:
[root@luminous-osc ~(keystone_admin)]# swift stat autodeleteContainer autodeletefile
Account: v1
Container: autodeleteContainer
Object: autodeletefile
Content Type: binary/octet-stream
Content Length: 71
Last Modified: Tue, 28 Aug 2018 08:31:37 GMT
ETag: c721f06d83ae2e2be3fa104ba7e79d13
Meta Mtime: 1535445040.199022
X-Delete-At: 1535445127
Accept-Ranges: bytes
X-Timestamp: 1535445097.44745
X-Trans-Id: tx00000000000000002f070-005b85087d-7e420-default
X-Openstack-Request-Id: tx00000000000000002f070-005b85087d-7e420-default
Check again after 30 seconds:
[root@luminous-osc ~(keystone_admin)]# swift stat autodeleteContainer autodeletefile
Object HEAD failed: http://106.120.114.157:8080/swift/v1/autodeleteContainer/autodeletefile 404 Not Found
Failed Transaction ID: tx00000000000000002f20b-005b850917-7e420-default
But the object is still present in the pool:
[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
342db702-df7e-4760-bacd-68341e78ede6.517152.3_autodeletefile
[root@luminous-osc ~(keystone_admin)]# radosgw-admin bucket list
[
"tempest-TestContainer-1849671326",
"tempest-TestContainer-1305170331",
"0eb72f67c1c74beca6f84432f8cc0ee9/autodeleteContainer"
]
Run the cleanup manually:
[root@luminous-osc ~(keystone_admin)]# radosgw-admin objects expire --debug-rgw=20
2018-08-28 17:37:29.827911 7f54d1c4b700 2 RGWDataChangesLog::ChangesRenewThread: start
...
2018-08-28 17:37:30.117161 7f54ecbeec40 15 got removal hint for: 1535445127 - 0eb72f67c1c74beca6f84432f8cc0ee9:autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3:autodeletefile:
2018-08-28 17:37:30.117222 7f54ecbeec40 20 reading from default.rgw.meta:root:.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3
2018-08-28 17:37:30.117237 7f54ecbeec40 20 get_system_obj_state: rctx=0x7ffcc14f53d0 obj=default.rgw.meta:root:.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 state=0x55d090ca7bb0 s->prefetch_data=0
2018-08-28 17:37:30.117243 7f54ecbeec40 10 cache get: name=default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 : miss
2018-08-28 17:37:30.118078 7f54ecbeec40 10 cache put: name=default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 info.flags=0x0
2018-08-28 17:37:30.118114 7f54ecbeec40 10 adding default.rgw.meta+root+.bucket.meta.autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3 to cache LRU end
2018-08-28 17:37:30.118121 7f54ecbeec40 15 NOTICE: cannot find bucket = autodeleteContainer. The object must be already removed
2018-08-28 17:37:30.118124 7f54ecbeec40 15 not actual hint for object: autodeletefile
2018-08-28 17:37:30.118135 7f54ecbeec40 20 trying to trim removal hints to=2018-08-28 17:37:29.887783, to_marker=1_1535445127.444034_0eb72f67c1c74beca6f84432f8cc0ee9:autodeleteContainer:342db702-df7e-4760-bacd-68341e78ede6.517152.3:autodeletefile:
...
Object is still present:
[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
342db702-df7e-4760-bacd-68341e78ede6.517152.3_autodeletefile
Object can be deleted via swift api but with the following error:
[root@luminous-osc ~(keystone_admin)]# swift delete autodeleteContainer autodeletefile
Error Deleting: autodeleteContainer/autodeletefile: Object DELETE failed: http://106.120.114.157:8080/swift/v1/autodeleteContainer/autodeletefile 404 Not Found NoSuchKey
[root@luminous-osc ~(keystone_admin)]# rados -p default.rgw.buckets.data ls
342db702-df7e-4760-bacd-68341e78ede6.4261.7_tempest-Object-499165745
History
#1 Updated by Jerzy Midura over 5 years ago
Important notice: on the same environment without
rgw keystone implicit tenants = true
The autodelete function is working fine.
#2 Updated by John Spray over 5 years ago
- Project changed from Ceph to rgw
#3 Updated by Abhishek Lekshmanan over 5 years ago
- Assignee set to Abhishek Lekshmanan