Project

General

Profile

Bug #47586

Able to circumvent S3 Object Lock using deleteobjects command

Added by Andrew Thomas over 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus, nautilus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Ceph supports the S3 object lock which prevents objects being deleted until their retention date has expired, however it appears to be possible to circumvent this and delete objects before their retention date has passed by using the option to delete multiple objects at once.

See the following example using the AWS CLI to confirm this.

Create an object
$ aws --profile=ceph --endpoint-url https://ceph s3api put-object --bucket andrew-test --key ceph-bug --object-lock-mode COMPLIANCE --object-lock-retain-until "2020-09-22 16:00:00" 
{
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "VersionId": "-cBhsp3jHdeWAgxH9Grd.ZCRHsyT8n-" 
}

Confirm object lock
$ aws --profile=ceph --endpoint-url https://ceph s3api get-object-retention --bucket andrew-test --key ceph-bug
{
    "Retention": {
        "Mode": "COMPLIANCE",
        "RetainUntilDate": "2020-09-22T16:00:00.000000000Z" 
    }
}

Attempt to delete the version using a single delete
$ aws --profile=ceph --endpoint-url https://ceph s3api delete-object --bucket andrew-test --key ceph-bug --version-id=-cBhsp3jHdeWAgxH9Grd.ZCRHsyT8n-

An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

This correctly fails.

Attempt to delete the version using a multiple delete
$ aws --profile=ceph --endpoint-url https://ceph s3api delete-objects --bucket andrew-test --delete "Objects=[{Key=ceph-bug,VersionId=-cBhsp3jHdeWAgxH9Grd.ZCRHsyT8n-}],Quiet=false" 
{
    "Deleted": [
        {
            "Key": "ceph-bug",
            "VersionId": "-cBhsp3jHdeWAgxH9Grd.ZCRHsyT8n-" 
        }
    ]
}

This succeeds and deletes the object version, despite it still being before the retention date.


Related issues

Related to rgw - Bug #59474: Cannot delete object using multi-delete operation on a bucket with policy New
Copied to rgw - Backport #48423: nautilus: Able to circumvent S3 Object Lock using deleteobjects command Resolved
Copied to rgw - Backport #48424: octopus: Able to circumvent S3 Object Lock using deleteobjects command Resolved

History

#1 Updated by Matt Benjamin over 3 years ago

  • Project changed from Ceph to rgw

#2 Updated by Aleksandr Rudenko over 3 years ago

Hi,

I can confirm this behavior on 14.2.11.

#4 Updated by Nathan Cutler over 3 years ago

  • Status changed from New to Fix Under Review
  • Backport set to octopus, nautilus
  • Pull request ID set to 37933

#5 Updated by J. Eric Ivancich over 3 years ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #48423: nautilus: Able to circumvent S3 Object Lock using deleteobjects command added

#7 Updated by Nathan Cutler over 3 years ago

  • Copied to Backport #48424: octopus: Able to circumvent S3 Object Lock using deleteobjects command added

#8 Updated by Simon Leinen almost 3 years ago

Isn't this bug more severe than "3 - minor"? It represents a clear violation of the promises of Object Lock. Backporting this to the Ceph versions actively used in the field (Nautilus and Octopus) would increase trust in Ceph again. For example, one of the ugly warnings in Veeam's object storage compatibility list1 could be eliminated:

Ceph (14.2.6 or later) [COMMUNITY ENTRY] POSSIBLE ISSUES (Date Format and >>> Immutability <<<)

(Sorry, I'm not personally involved in running the Ceph systems here, so I won't be able to backport or test the fix myself.)

[1] https://forums.veeam.com/object-storage-f52/unoffizial-compatibility-list-for-veeam-cloud-tier-t56956.html

#9 Updated by Robert Groenewald almost 3 years ago

This issue needs to picked up! This makes the reason why we are running Ceph almost unusable!

#10 Updated by Matt Benjamin almost 3 years ago

  • Assignee set to Matt Benjamin

#11 Updated by Matt Benjamin almost 3 years ago

  • Severity changed from 3 - minor to 2 - major

(for bug scrub 4/29: I will take ownership of this issue)

#12 Updated by Simon Leinen almost 3 years ago

Matt Benjamin wrote:

(for bug scrub 4/29: I will take ownership of this issue)

Thank you, Matt!

#13 Updated by Loïc Dachary almost 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

#14 Updated by J. Eric Ivancich 11 months ago

  • Related to Bug #59474: Cannot delete object using multi-delete operation on a bucket with policy added

Also available in: Atom PDF