Bug #63537
openobject lock: The query does not match after the worm protection time is set to 2124, 1, 1
0%
Description
Set the protection time to 2124.1.1 and then check the protection of this object to get 198x.x.x time
res=client1.copy_object(Bucket=dst_bucket, Key=dst_object,CopySource=str(src_bucket+'/'+src_object),ObjectLockMode='GOVERNANCE',ObjectLockRetainUntilDate=datetime(2124, 1, 1),ObjectLockLegalHoldStatus='ON')
resp = client1.get_object_retention(Bucket=dst_bucket,Key=dst_object)
print(resp['Retention'])
resp = client1.get_object_legal_hold(Bucket=dst_bucket,Key=dst_object)
print(resp['LegalHold'])
The output after executing the python script is
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(1987, 11, 24, 17, 31, 44, tzinfo=tzlocal())}
{'Status': 'ON'}
Updated by Casey Bodley 6 months ago
- Related to Bug #56993: rgw_object_lock.cc:The maximum time of bucket object lock is 24855 days added
Updated by Casey Bodley 6 months ago
- Subject changed from The query does not match after the worm protection time is set to 2124, 1, 1 to object lock: The query does not match after the worm protection time is set to 2124, 1, 1
- Tags set to object-lock
Updated by Casey Bodley 6 months ago
- Status changed from New to Triaged
i was able to reproduce this by modifying the following s3-tests case:
~/s3-tests $ git diff
diff --git a/s3tests_boto3/functional/test_s3.py b/s3tests_boto3/functional/test_s3.py
index 1273d8a..8a2c501 100644
--- a/s3tests_boto3/functional/test_s3.py
+++ b/s3tests_boto3/functional/test_s3.py
@@ -11776,9 +11776,11 @@ def test_object_lock_put_obj_retention():
key = 'file1'
response = client.put_object(Bucket=bucket_name, Body='abc', Key=key)
version_id = response['VersionId']
- retention = {'Mode':'GOVERNANCE', 'RetainUntilDate':datetime.datetime(2030,1,1,tzinfo=pytz.UTC)}
+ retention = {'Mode':'GOVERNANCE', 'RetainUntilDate':datetime.datetime(2141,1,1,tzinfo=pytz.UTC)}
response = client.put_object_retention(Bucket=bucket_name, Key=key, Retention=retention)
assert response['ResponseMetadata']['HTTPStatusCode'] == 200
+ response = client.get_object_retention(Bucket=bucket_name, Key=key)
+ assert response['Retention'] == retention
client.delete_object(Bucket=bucket_name, Key=key, VersionId=version_id, BypassGovernanceRetention=True)
> assert response['Retention'] == retention E AssertionError: assert {'Mode': 'GOV...info=tzutc())} == {'Mode': 'GOV...tzinfo=<UTC>)} E Omitting 1 identical items, use -vv to show E Differing items: E {'RetainUntilDate': datetime.datetime(2004, 11, 24, 17, 31, 44, tzinfo=tzutc())} != {'RetainUntilDate': datetime.datetime(2141, 1, 1, 0, 0, tzinfo=< UTC>)}
debug logging shows that ceph::from_iso_8601()
parses the date correctly, but binary encode/decode corrupts the date:
RetainUntilDate "2141-01-01T00:00:00Z" parsed as 2140-12-31T19:00:00.000000-0500 encoding retain_until_date 2140-12-31T19:00:00.000000-0500 ... decoded retain_until_date 2004-11-24T12:31:44.000000-0500 RetainUntilDate "2004-11-24T12:31:44.000000-0500" formatted as 2004-11-24T17:31:44.000000000Z
it looks like the binary encoding for these timestamps is truncating seconds to 32 bits in https://github.com/ceph/ceph/blob/d9a4752/src/include/encoding.h#L332-L336:
// A 32 bit count of seconds causes me vast unhappiness.
uint32_t s = ts.tv_sec;
Updated by Casey Bodley 6 months ago
- Status changed from Triaged to Fix Under Review
- Assignee set to Casey Bodley
- Backport set to quincy reef
- Pull request ID set to 54516
Updated by djf daijufang 6 months ago
Here's another way to solve the problem, set the retain_until_date to a 64-bit integer variable.
https://github.com/ceph/ceph/pull/54585
The test results of this method are shown below.
When the expiration time is set to 2124,1,1, the result of running the test script is as follows:
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2124, 1, 1, 0, 0, tzinfo=tzlocal())}
{'Status': 'ON'}
When the expiration time is set to 2144,1,1, the result of running the test script is as follows:
{'Mode': 'GOVERNANCE', 'RetainUntilDate': datetime.datetime(2144, 1, 1, 0, 0, tzinfo=tzlocal())}
{'Status': 'ON'}
After the modification, the expiration time is displayed normally.
The test script is:
def test_object_lock():
src_bucket=bucket_noworm
dst_bucket=bucket_worm
client1.create_bucket(Bucket=dst_bucket, ObjectLockEnabledForBucket=True)
client1.put_object_lock_configuration(Bucket=dst_bucket, ObjectLockConfiguration={'ObjectLockEnabled': 'Enabled'})
src_object=prefix
dst_object=prefix+"dst5"
client1.put_object(Bucket=src_bucket, Key=src_object, Body=open(path + "/" + "10M", 'rb').read())
res=client1.copy_object(Bucket=dst_bucket, Key=dst_object,CopySource=str(src_bucket+'/'+src_object),ObjectLockMode='GOVERNANCE',ObjectLockRetainUntilDate=datetime(2124, 1, 1),ObjectLockLegalHoldStatus='ON')
resp = client1.get_object_retention(Bucket=dst_bucket,Key=dst_object)
print(resp['Retention'])
resp = client1.get_object_legal_hold(Bucket=dst_bucket,Key=dst_object)
print(resp['LegalHold'])
Updated by Casey Bodley 5 months ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 5 months ago
- Copied to Backport #63627: quincy: object lock: The query does not match after the worm protection time is set to 2124, 1, 1 added
Updated by Backport Bot 5 months ago
- Copied to Backport #63628: reef: object lock: The query does not match after the worm protection time is set to 2124, 1, 1 added
Updated by Backport Bot 5 months ago
- Tags changed from object-lock to object-lock backport_processed