Ceph » rgw

aws s3api get-object-lock-configuration --bucket locktest --endpoint http://our-s3

{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 50
}
}
}
}

Hi Folks,

Working against current ceph/master, I believe that I have successfully verified the main workflows for S3 object lock, with retention policy.

The key observation that must be made up front is that S3 object lock is built on the foundation of S3 object versioning. S3 object lock retention policies, as shown below, protect object versions, not key names. What this means is, deleting a key name associated with a protected object version, is not a violation of the retention policy. Nor is overwriting the key name with a new version. Only attempts to remove a protected version actually violate the contract of S3 object lock. The example workflows below demonstrate this, in two variations: 1) with a default retention policy (all object versions are protected) 2) without a default retention policy (only explicitly protected object versions are protected). I have not separately verified legal hold nor governance exceptions.

N.b., in the process of testing, I noticed that the lack of client-visible error messages made it harder to understand what object lock was doing. To address this, I've made an upstream PR which returns human-readable errors when several object lock operations fail.

regards,

Matt

aws cli transcript follows:
# create bucket for retention
aws --endpoint http://lemon:8000 s3api create-bucket --bucket foopio --object-lock-enabled-for-bucket 

# set default retention policy
 aws --endpoint http://lemon:8000 s3api put-object-lock-configuration --bucket foopio --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 10 }}}'

# store object--this should inherit default policy?
aws --endpoint http://lemon:8000 s3api put-object --bucket foopio --key testc1 --body /etc/passwd
{
    "ETag": "\"f828d3ae09a4fef58bf70af6cf59fde4\"",
    "VersionId": "ir-eMeQAWEfQu6M48kYpEMNo8aqucFi" 
}

# check version(s)
aws --endpoint http://lemon:8000 s3api list-object-versions --bucket foopio
{
    "Versions": [
        {
            "ETag": "\"f828d3ae09a4fef58bf70af6cf59fde4\"",
            "Size": 4411,
            "StorageClass": "STANDARD",
            "Key": "testc1",
            "VersionId": "ir-eMeQAWEfQu6M48kYpEMNo8aqucFi",
            "IsLatest": true,
            "LastModified": "2021-02-28T22:05:32.812000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ]
}

# try to overwrite
aws --endpoint http://lemon:8000 s3api put-object --bucket foopio --key testc1 --body /etc/group
{
    "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
    "VersionId": "tS5EFUBlzqrdH29CGx8DMqKZuiPjxuF" 
}

aws --endpoint http://lemon:8000 s3api list-object-versions --bucket foopio

{
    "Versions": [
        {
            "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
            "Size": 1639,
            "StorageClass": "STANDARD",
            "Key": "testc1",
            "VersionId": "tS5EFUBlzqrdH29CGx8DMqKZuiPjxuF",
            "IsLatest": true,
            "LastModified": "2021-02-28T22:10:33.143000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        },
        {
            "ETag": "\"f828d3ae09a4fef58bf70af6cf59fde4\"",
            "Size": 4411,
            "StorageClass": "STANDARD",
            "Key": "testc1",
            "VersionId": "ir-eMeQAWEfQu6M48kYpEMNo8aqucFi",
            "IsLatest": false,
            "LastModified": "2021-02-28T22:05:32.812000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ]
}

# try to delete protected version (all versions are protected!)
aws --endpoint http://lemon:8000 s3api delete-object --bucket foopio --key testc1  --version-id 'tS5EFUBlzqrdH29CGx8DMqKZuiPjxuF'
An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

aws --endpoint http://lemon:8000 s3api delete-object --bucket foopio --key testc1  --version-id 'ir-eMeQAWEfQu6M48kYpEMNo8aqucFi'

An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

(with improved error reporting:
An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock)

# workflow without a default retention policy--same as above, except,
  only explicitly configured object versions are protected

[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api
create-bucket --bucket katrina --object-lock-enabled-for-bucket

#create object testc1 with no protection
[mbenjamin@lemon python]$ aws --endpoint http://lemon:8000 s3api put-object --bucket katrina --key testc1 --body /etc/group
{
    "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
    "VersionId": "ep.vH5DgTW9-9glSaZSlf6vSKmOVfq1" 
}
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api delete-object --bucket katrina --key testc1  --version-id 'ep.vH5DgTW9-9glSaZSlf6vSKmOVfq1'
{
    "VersionId": "ep.vH5DgTW9-9glSaZSlf6vSKmOVfq1" 
}
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api list-object-versions --bucket katrina
[mbenjamin@lemon python]$
# testc1 deleted (correct result)

# create object  testc2
[mbenjamin@lemon python]$ aws --endpoint http://lemon:8000 s3api put-object --bucket katrina --key testc2 --body /etc/group
{
    "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
    "VersionId": "p9vm-eeO2FYVIKO2pv1692NCYaKuDxU" 
}

# and set retention policy
aws --endpoint http://lemon:8000 s3api put-object-retention --bucket katrina --key testc2 --retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2025-01-01T00:00:00" }'

# store new, unprotected version at key testc2
[mbenjamin@lemon python]$ aws --endpoint http://lemon:8000 s3api put-object --bucket katrina --key testc2 --body /etc/fstab
{
    "ETag": "\"00bbd9047796e4b1b2577902ccaeb4a0\"",
    "VersionId": "mYUI8-Cqs8pK7rfVHlnBdH.gpLuwTx0" 
}
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api list-object-versions --bucket katrina
{
    "Versions": [
        {
            "ETag": "\"00bbd9047796e4b1b2577902ccaeb4a0\"",
            "Size": 515,
            "StorageClass": "STANDARD",
            "Key": "testc2",
            "VersionId": "mYUI8-Cqs8pK7rfVHlnBdH.gpLuwTx0",
            "IsLatest": true,
            "LastModified": "2021-02-28T22:57:41.954000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        },
        {
            "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
            "Size": 1639,
            "StorageClass": "STANDARD",
            "Key": "testc2",
            "VersionId": "p9vm-eeO2FYVIKO2pv1692NCYaKuDxU",
            "IsLatest": false,
            "LastModified": "2021-02-28T22:55:09.228000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ]
}
# (yes, the key "testc2" can be overwritten--but the protected version
#has not been damaged)

# try to delete
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api delete-object --bucket katrina --key testc2  --version-id 'p9vm-eeO2FYVIKO2pv1692NCYaKuDxU'

An error occurred (AccessDenied) when calling the DeleteObject operation: forbidden by object lock

# we can delete the unprotected one, though
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api delete-object --bucket katrina --key testc2  --version-id 'mYUI8-Cqs8pK7rfVHlnBdH.gpLuwTx0'
{
    "VersionId": "mYUI8-Cqs8pK7rfVHlnBdH.gpLuwTx0" 
}
[mbenjamin@lemon python]$  aws --endpoint http://lemon:8000 s3api list-object-versions --bucket katrina
{
    "Versions": [
        {
            "ETag": "\"4333a95e808398f4b059fced5276ca91\"",
            "Size": 1639,
            "StorageClass": "STANDARD",
            "Key": "testc2",
            "VersionId": "p9vm-eeO2FYVIKO2pv1692NCYaKuDxU",
            "IsLatest": true,
            "LastModified": "2021-02-28T22:55:09.228000+00:00",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ]
}