Bug #63724
openobject lock: An object uploaded through a multipart upload can be deleted without the x-amz-bypass-governance-retention header
0%
Description
Set object locks on buckets and upload objects to buckets. Objects can be deleted without the x-amz-bypass-governance-retention header.
Set object locks on buckets and upload objects.
from datetime import datetime
def _add_header(request, **kwargs):
request.headers.add_header('x-amz-bypass-governance-retention', "true")
event_system = s3_client.meta.events
event_system.register_first('before-sign.s3.*', _add_header)
s3_client.create_bucket(Bucket=bucketname, ObjectLockEnabledForBucket=True)
s3_client.put_object_lock_configuration(
Bucket=bucketname,
ObjectLockConfiguration={
'ObjectLockEnabled': 'Enabled',
'Rule': {
'DefaultRetention': {
'Mode': 'GOVERNANCE', # COMPLIANCE || GOVERNANCE
'Days': 1,
# 'Years':123
}
}
}
)
mpu = s3_client.create_multipart_upload(Bucket=bucketname,Key=objectname,ObjectLockLegalHoldStatus='ON')
uploadid = mpu["UploadId"]
part_info = {'Parts': []}
res = s3_client.upload_part(Bucket=bucketname, Key=objectname, PartNumber=1, UploadId=uploadid, Body="a"*1024*3)
part_info['Parts'].append({'PartNumber': 1, 'ETag': res['ETag']})
s3_client.complete_multipart_upload(Bucket=bucketname, Key=objectname, UploadId=uploadid, MultipartUpload=part_info)
resp = s3_client.get_object_retention(Bucket=bucketname, Key=objectname)
print(resp['Retention'])
The result of the script is as follows.
@5257f20a581e ▶ python3 worm_test.py
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2023, 12, 5, 7, 16, 56, 655410, tzinfo=tzlocal())}
@5257f20a581e ▶ s3cmd ls s3://buck-lock
2023-12-04 08:32 3072 s3://buck-lock/50M
Deletes the object without a BypassGovernanceRetention header.
resp = s3_client.get_object_retention(Bucket=bucketname, Key=objectname)
print(resp['Retention'])
s3_client.delete_object(Bucket=bucketname, Key=objectname)
The result of the script is as follows. Even if object locks are set on the bucket, objects uploaded through multipart upload can still be deleted
@5257f20a581e ▶ python3 worm_test2.py
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2023, 12, 5, 7, 16, 56, 655410, tzinfo=tzlocal())}
…/xx/ceph/build loacl_ceph via △ v3.20.2 v3.9.18 took 2s
@5257f20a581e ▶ s3cmd ls s3://buck-lock
…/xx/ceph/build loacl_ceph via △ v3.20.2 v3.9.18
@5257f20a581e ▶
Updated by djf daijufang 5 months ago
The fix is been submitted to the link https://github.com/ceph/ceph/pull/54767
The test results are shown below.
The request to delete an object does not contain the x-amz-bypass-governance-retention header. The test script is as follow.
#!/usr/bin/python
from boto3.session import Session
import boto3
import logging
import os,sys
bucketname= 'buck-lock2'
objectname= '50M'
access_key = 'user1'
secret_key = 'user1'
endpoint = 'http://127.0.0.1:8000'
session = Session(access_key, secret_key)
config = boto3.session.Config(connect_timeout=3000000, read_timeout=3000000, retries={'max_attempts': 0}, signature_version="s3")
session = Session(access_key, secret_key)
s3_client = session.client('s3', endpoint_url=endpoint, config=config)
resp = s3_client.get_object_retention(Bucket=bucketname, Key=objectname)
print(resp['Retention'])
s3_client.delete_object(Bucket=bucketname, Key=objectname)
The following is the result of the script. Objects cannot be deleted without the x-amz-bypass-governance-retention header, returning a 403 error.
root@18d470fd3a9d ▶ python3 worm_multi.py
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2023, 12, 5, 3, 28, 5, tzinfo=tzlocal())}
Traceback (most recent call last):
File "/home/djf/ceph/build/worm_multi.py", line 72, in <module>
s3_client.delete_object(Bucket=bucketname, Key=objectname, BypassGovernanceRetention=True, VersionId='WMwjhMNcByR6Sc-CC7HlmrkHwB5XCxh')# BypassGovernanceRetention=True,
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock
The request to delete an object contains the x-amz-bypass-governance-retention header. The test script is as follow.
resp = s3_client.get_object_retention(Bucket=bucketname, Key=objectname)
print(resp['Retention'])
esp = s3_client.put_object_legal_hold(Bucket=bucketname, Key=objectname, LegalHold={'Status': 'OFF'})
resp = s3_client.get_object_legal_hold(Bucket=bucketname,Key=objectname)
print(resp)
s3_client.delete_object(Bucket=bucketname, Key=objectname, BypassGovernanceRetention=True)
The script is executed as follows. In the case with the x-amz-bypass-governance-retention header and with the LegalHold equal oFF, the object is successfully deleted.
root@18d470fd3a9d ▶ python3 worm_multi.py
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2023, 12, 5, 11, 31, 45, tzinfo=tzlocal())}
{'ResponseMetadata': {'RequestId': 'tx0000077d81a1239d0c405-00656db8c7-1196-default', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-request-id': 'tx0000077d81a1239d0c405-00656db8c7-1196-default', 'content-type': 'application/xml', 'server': 'Ceph Object Gateway (reef)', 'content-length': '81', 'date': 'Mon, 04 Dec 2023 11:32:23 GMT', 'connection': 'Keep-Alive'}, 'RetryAttempts': 0}, 'LegalHold': {'Status': 'OFF'}}
Updated by Casey Bodley 5 months ago
- Status changed from New to Fix Under Review
- Tags set to object-lock
- Backport set to quincy reef
- Pull request ID set to 54767
Updated by Casey Bodley 3 months ago
- Status changed from Fix Under Review to Need More Info
i had trouble reproducing this on main with the tests in https://github.com/ceph/s3-tests/pull/547
Updated by Casey Bodley 3 months ago
- Status changed from Need More Info to Fix Under Review
- Backport changed from quincy reef to quincy reef squid
Updated by Casey Bodley 2 months ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 2 months ago
- Copied to Backport #64664: squid: object lock: An object uploaded through a multipart upload can be deleted without the x-amz-bypass-governance-retention header added
Updated by Backport Bot 2 months ago
- Copied to Backport #64665: quincy: object lock: An object uploaded through a multipart upload can be deleted without the x-amz-bypass-governance-retention header added
Updated by Backport Bot 2 months ago
- Copied to Backport #64666: reef: object lock: An object uploaded through a multipart upload can be deleted without the x-amz-bypass-governance-retention header added
Updated by Backport Bot 2 months ago
- Tags changed from object-lock to object-lock backport_processed