Project

General

Profile

Bug #43226

rgw: object version can be deleted without TOTP on bucket that has MFA Delete enabled.

Added by Alfonso Martínez 10 months ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

It seems that an object version can be deleted without TOTP on bucket that has MFA Delete enabled.
- Environment: CentOS 7 + ceph octopus (master) installed through rpm packages built in shaman.
- TOTP generator: FreeOTP (Android).

[root@ceph-rpm ceph]# ceph -v
ceph version 15.0.0-8192-gb976dc5 (b976dc5aa33344fd4736a8ae3a4dce9d4351864d) octopus (dev)

[root@ceph-rpm ceph]# radosgw-admin mfa create --uid=dev --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32

[root@ceph-rpm ceph]# alias aws="aws --endpoint-url=http://localhost:8000" 

[root@ceph-rpm ceph]# aws s3api create-bucket --bucket test

[root@ceph-rpm ceph]# aws s3api list-buckets
{
    "Owner": {
        "DisplayName": "Dev Admin", 
        "ID": "dev" 
    }, 
    "Buckets": [
        {
            "CreationDate": "2019-12-10T11:49:53.781Z", 
            "Name": "test" 
        }
    ]
}

[root@ceph-rpm ceph]# aws s3api get-bucket-versioning --bucket test
[root@ceph-rpm ceph]#

[root@ceph-rpm ceph]# aws s3api put-bucket-versioning --bucket test --versioning-configuration '{"Status":"Enabled","MFADelete":"Enabled"}' --mfa '1 221402'

[root@ceph-rpm ceph]# aws s3api get-bucket-versioning --bucket test
{
    "Status": "Enabled", 
    "MFADelete": "Enabled" 
}

[root@ceph-rpm ceph]# aws s3api put-object --bucket test --key example --body CONTRIBUTING.rst
{
    "VersionId": "ZrRv3hX0CgbjNo9j4egnexvFTlPa--x", 
    "ETag": "\"2551b46bd421838b7a5fca325f12818c\"" 
}

[root@ceph-rpm ceph]# aws s3api list-object-versions --bucket test --key example
{
    "Name": "test", 
    "Versions": [
        {
            "LastModified": "2019-12-10T12:12:41.776Z", 
            "VersionId": "ZrRv3hX0CgbjNo9j4egnexvFTlPa--x", 
            "ETag": "\"2551b46bd421838b7a5fca325f12818c\"", 
            "StorageClass": "STANDARD", 
            "Key": "example", 
            "Owner": {
                "DisplayName": "Dev Admin", 
                "ID": "dev" 
            }, 
            "IsLatest": true, 
            "Size": 640
        }
    ], 
    "MaxKeys": 1000, 
    "Prefix": "", 
    "KeyMarker": "example", 
    "IsTruncated": false, 
    "VersionIdMarker": "" 
}

[root@ceph-rpm ceph]# aws s3api delete-object --bucket test --key example --version-id ZrRv3hX0CgbjNo9j4egnexvFTlPa--x
{
    "VersionId": "ZrRv3hX0CgbjNo9j4egnexvFTlPa--x" 
}
[root@ceph-rpm ceph]# aws s3api list-object-versions --bucket test --key example
{
    "MaxKeys": 1000, 
    "Prefix": "", 
    "Name": "test", 
    "KeyMarker": "example", 
    "IsTruncated": false, 
    "VersionIdMarker": "" 
}

History

#1 Updated by Alfonso Martínez 10 months ago

  • Subject changed from rgw: object version can be deleted on bucket that has MFA Delete enabled. to rgw: object version can be deleted without TOTP on bucket that has MFA Delete enabled.
  • Description updated (diff)

#2 Updated by Sage Weil 10 months ago

  • Project changed from mgr to rgw

#3 Updated by Matthew Oliver 10 months ago

Pardon my ignorance, but wouldn't you be authenticating with:

[root@ceph-rpm ceph]# aws s3api put-bucket-versioning --bucket test --versioning-configuration '{"Status":"Enabled","MFADelete":"Enabled"}' --mfa '1 221402'

I think the default duration of a lease is an hour 3600 seconds according to rgw_sts.h:

static constexpr uint64_t DEFAULT_DURATION_IN_SECS = 3600;

Was the delete command issued within that timeframe. Or is s3 with MFADelete enabled expect you to pass an --mfa <..> each time?

I'm new to the RGW codebase and to be honest to s3's TOTP. Just want to make sure before I attempt to delve in. I guess I can at least see if I can recreate it.

#4 Updated by Matthew Oliver 10 months ago

OK, I've tested this on my dev env, and, of course I was wrong :)
You can't delete a version unless you supply the '--mfa "<serial> "'. Having said that, it's working fine in my master branch on OpenSuse Leap 15.1. I haven't pulled in a while so let me go check the latest code.

#5 Updated by Matthew Oliver 10 months ago

I just built todays master, and I can't reproduce. NOTE that I am building from source (do_cmake, make and vstart).

#6 Updated by Alfonso Martínez 10 months ago

Matthew Oliver wrote:

I just built todays master, and I can't reproduce. NOTE that I am building from source (do_cmake, make and vstart).

I was able to reproduce it with ceph installed through rpm packages,
Maybe the issue can be related to the rpm build process.
Can you test it that way?

#7 Updated by Matthew Oliver 9 months ago

I'll setup a centos env and first test it there. I'll also build a new OpenSuse env and test it with the rpms, so attempt to tackle it from every angle.

#8 Updated by Matthew Oliver 9 months ago

Sorry for the lateness of my reply. I've gone down a few rabbit holes. Firstly I deployed a centos 7 cloud image, build gcc so I could build the source. The source isn't building, maybe because of rocksdb version.. anyway.. still going there.

At the same time, I created another centos 7 cloudvm, added the shaman repo for master. Installed the ceph components from this repo. Next, I hacked up a vstart.sh and stop.sh to work with installed ceph (see my junk scripts ceph folder: https://github.com/matthewoliver/junk/tree/master/ceph )

Then have attempted what you did:

[centos@moliver-ceph2 build]$ radosgw-admin mfa create --uid=testid --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32
2019-12-19T05:42:16.171+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.
2019-12-19T05:42:16.203+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.
2019-12-19T05:42:16.209+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.

[centos@moliver-ceph2 build]$ alias aws="aws --endpoint-url=http://localhost:8000" 

[centos@moliver-ceph2 build]$ aws s3api create-bucket --bucket test

[centos@moliver-ceph2 build]$ aws s3api list-buckets
{
    "Buckets": [
        {
            "Name": "test",
            "CreationDate": "2019-12-19T05:42:39.806Z" 
        }
    ],
    "Owner": {
        "DisplayName": "M. Tester",
        "ID": "testid" 
    }
}

[centos@moliver-ceph2 build]$ aws s3api put-bucket-versioning --bucket test --versioning-configuration '{"Status":"Enabled","MFADelete":"Enabled"}' --mfa '1 863726'

[centos@moliver-ceph2 build]$ aws s3api get-bucket-versioning --bucket test
{
    "Status": "Enabled",
    "MFADelete": "Enabled" 
}

[centos@moliver-ceph2 build]$ aws s3api put-object --bucket test --key example --body vstart_environment.sh 
{
    "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
    "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa" 
}

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Versions": [
        {
            "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
            "Size": 125,
            "StorageClass": "STANDARD",
            "Key": "example",
            "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa",
            "IsLatest": true,
            "LastModified": "2019-12-19T05:44:19.237Z",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ],
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id 2sAKsln0m5TIMVLw583d-i6JlUbhjQa

An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id 2sAKsln0m5TIMVLw583d-i6JlUbhjQa --mfa '1 005223'
{
    "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa" 
}

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

As you can see, it's successfully gives me an access denied.

#9 Updated by Matthew Oliver 9 months ago

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

#10 Updated by Alfonso Martínez 9 months ago

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:
"Owner": {
"DisplayName": "Dev Admin",
"ID": "dev"
},

and created an mfa token for him:

radosgw-admin mfa create --uid=dev --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32

I also see that when you list object versions, your object "example" has a property that was not listed on mine:
"EncodingType": "url"

What ceph version are you using?

#11 Updated by Casey Bodley 9 months ago

Alfonso MH wrote:

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:

note that --system and --admin users override permission checks, so you should avoid them for general use

#12 Updated by Casey Bodley 9 months ago

  • Status changed from New to Triaged

#13 Updated by Matthew Oliver 9 months ago

Alfonso MH wrote:

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:
"Owner": {
"DisplayName": "Dev Admin",
"ID": "dev"
},

and created an mfa token for him:

radosgw-admin mfa create --uid=dev --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32

I also see that when you list object versions, your object "example" has a property that was not listed on mine:
"EncodingType": "url"

What ceph version are you using?

[centos@moliver-ceph2 ceph]$ ceph --version
ceph version 15.0.0-8545-gb31a020 (b31a0200089df0682847b3ab732bb06b505b9d2c) octopus (dev)

Let me go and try creating a user with --system privs, just in case that's it. As Alfonso mentioned it might be because of this it's bypassing the prem checks.

Have to run out to dinner now, but will check as soon as I can.

#14 Updated by Matthew Oliver 9 months ago

Yup, success! I can confirm I can repeat what your seeing when I use a user created with `--system`:

[centos@moliver-ceph2 build]$ radosgw-admin user create --uid matt --display-name 'matt oliver' --system

...

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Versions": [
        {
            "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
            "Size": 125,
            "StorageClass": "STANDARD",
            "Key": "example",
            "VersionId": "xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h",
            "IsLatest": true,
            "LastModified": "2019-12-20T10:07:22.071Z",
            "Owner": {
                "DisplayName": "matt oliver",
                "ID": "matt" 
            }
        }
    ],
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}
[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h
{
    "VersionId": "xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h" 
}
[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

So I guess now the real question, seeing as Casey wrote:

note that --system and --admin users override permission checks, so you should avoid them for general use

`--system` override permission checks, this might actually be the correct and expected behaviour? If so, the bare minimum to fix this bug I'd suspect is to confirm and make sure that this behaviour is documented for TOTP in RGW.

If it isn't the right behaviour then I that's something that we obviously fix.. I'm probably too new to make that call, maybe we need to take it to the community.

#15 Updated by Casey Bodley 9 months ago

  • Status changed from Triaged to Closed

closing as not-a-bug, thanks for the discussion!

Also available in: Atom PDF