Ceph » rgw

Pardon my ignorance, but wouldn't you be authenticating with:

[root@ceph-rpm ceph]# aws s3api put-bucket-versioning --bucket test --versioning-configuration '{"Status":"Enabled","MFADelete":"Enabled"}' --mfa '1 221402'

I think the default duration of a lease is an hour 3600 seconds according to rgw_sts.h:

static constexpr uint64_t DEFAULT_DURATION_IN_SECS = 3600;

Was the delete command issued within that timeframe. Or is s3 with MFADelete enabled expect you to pass an --mfa <..> each time?

I'm new to the RGW codebase and to be honest to s3's TOTP. Just want to make sure before I attempt to delve in. I guess I can at least see if I can recreate it.

OK, I've tested this on my dev env, and, of course I was wrong :)
You can't delete a version unless you supply the '--mfa "<serial> "'. Having said that, it's working fine in my master branch on OpenSuse Leap 15.1. I haven't pulled in a while so let me go check the latest code.

I just built todays master, and I can't reproduce. NOTE that I am building from source (do_cmake, make and vstart).

Matthew Oliver wrote:

I just built todays master, and I can't reproduce. NOTE that I am building from source (do_cmake, make and vstart).

I was able to reproduce it with ceph installed through rpm packages,
Maybe the issue can be related to the rpm build process.
Can you test it that way?

I'll setup a centos env and first test it there. I'll also build a new OpenSuse env and test it with the rpms, so attempt to tackle it from every angle.

Sorry for the lateness of my reply. I've gone down a few rabbit holes. Firstly I deployed a centos 7 cloud image, build gcc so I could build the source. The source isn't building, maybe because of rocksdb version.. anyway.. still going there.

At the same time, I created another centos 7 cloudvm, added the shaman repo for master. Installed the ceph components from this repo. Next, I hacked up a vstart.sh and stop.sh to work with installed ceph (see my junk scripts ceph folder: https://github.com/matthewoliver/junk/tree/master/ceph )

Then have attempted what you did:

[centos@moliver-ceph2 build]$ radosgw-admin mfa create --uid=testid --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32
2019-12-19T05:42:16.171+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.
2019-12-19T05:42:16.203+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.
2019-12-19T05:42:16.209+0000 7f8bbc9d9ec0 -1 WARNING: all dangerous and experimental features are enabled.

[centos@moliver-ceph2 build]$ alias aws="aws --endpoint-url=http://localhost:8000" 

[centos@moliver-ceph2 build]$ aws s3api create-bucket --bucket test

[centos@moliver-ceph2 build]$ aws s3api list-buckets
{
    "Buckets": [
        {
            "Name": "test",
            "CreationDate": "2019-12-19T05:42:39.806Z" 
        }
    ],
    "Owner": {
        "DisplayName": "M. Tester",
        "ID": "testid" 
    }
}

[centos@moliver-ceph2 build]$ aws s3api put-bucket-versioning --bucket test --versioning-configuration '{"Status":"Enabled","MFADelete":"Enabled"}' --mfa '1 863726'

[centos@moliver-ceph2 build]$ aws s3api get-bucket-versioning --bucket test
{
    "Status": "Enabled",
    "MFADelete": "Enabled" 
}

[centos@moliver-ceph2 build]$ aws s3api put-object --bucket test --key example --body vstart_environment.sh 
{
    "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
    "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa" 
}

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Versions": [
        {
            "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
            "Size": 125,
            "StorageClass": "STANDARD",
            "Key": "example",
            "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa",
            "IsLatest": true,
            "LastModified": "2019-12-19T05:44:19.237Z",
            "Owner": {
                "DisplayName": "M. Tester",
                "ID": "testid" 
            }
        }
    ],
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id 2sAKsln0m5TIMVLw583d-i6JlUbhjQa

An error occurred (AccessDenied) when calling the DeleteObject operation: Unknown

[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id 2sAKsln0m5TIMVLw583d-i6JlUbhjQa --mfa '1 005223'
{
    "VersionId": "2sAKsln0m5TIMVLw583d-i6JlUbhjQa" 
}

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

As you can see, it's successfully gives me an access denied.

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:
"Owner": {
"DisplayName": "Dev Admin",
"ID": "dev"
},

and created an mfa token for him:

radosgw-admin mfa create --uid=dev --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32

I also see that when you list object versions, your object "example" has a property that was not listed on mine:
"EncodingType": "url"

What ceph version are you using?

Alfonso MH wrote:

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:

note that --system and --admin users override permission checks, so you should avoid them for general use

Alfonso MH wrote:

Matthew Oliver wrote:

Any other hints on what you've done differently that might be the root cause? Because I want to be able to recreate it.

I'm using the default vstart user. Maybe you create your user in some different way? I dunno, tenanted? Just trying to figure out what's different.

Yes, in my example I created another use called "dev" with "--system" privileges:
"Owner": {
"DisplayName": "Dev Admin",
"ID": "dev"
},

and created an mfa token for him:

radosgw-admin mfa create --uid=dev --totp-serial=1 --totp-seed=23456723 --totp-seed-type=base32

I also see that when you list object versions, your object "example" has a property that was not listed on mine:
"EncodingType": "url"

What ceph version are you using?

[centos@moliver-ceph2 ceph]$ ceph --version
ceph version 15.0.0-8545-gb31a020 (b31a0200089df0682847b3ab732bb06b505b9d2c) octopus (dev)

Let me go and try creating a user with --system privs, just in case that's it. As Alfonso mentioned it might be because of this it's bypassing the prem checks.

Have to run out to dinner now, but will check as soon as I can.

Yup, success! I can confirm I can repeat what your seeing when I use a user created with `--system`:

[centos@moliver-ceph2 build]$ radosgw-admin user create --uid matt --display-name 'matt oliver' --system

...

[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Versions": [
        {
            "ETag": "\"6f3d6c2c0684b7fa33c56eaa2559da63\"",
            "Size": 125,
            "StorageClass": "STANDARD",
            "Key": "example",
            "VersionId": "xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h",
            "IsLatest": true,
            "LastModified": "2019-12-20T10:07:22.071Z",
            "Owner": {
                "DisplayName": "matt oliver",
                "ID": "matt" 
            }
        }
    ],
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}
[centos@moliver-ceph2 build]$ aws s3api delete-object --bucket test --key example --version-id xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h
{
    "VersionId": "xaZpp.CyZqJKCo9QeXSqp7U2h.RFy-h" 
}
[centos@moliver-ceph2 build]$ aws s3api list-object-versions --bucket test --key example
{
    "IsTruncated": false,
    "KeyMarker": "example",
    "VersionIdMarker": "",
    "Name": "test",
    "Prefix": "",
    "MaxKeys": 1000,
    "EncodingType": "url" 
}

So I guess now the real question, seeing as Casey wrote:

note that --system and --admin users override permission checks, so you should avoid them for general use

`--system` override permission checks, this might actually be the correct and expected behaviour? If so, the bare minimum to fix this bug I'd suspect is to confirm and make sure that this behaviour is documented for TOTP in RGW.

If it isn't the right behaviour then I that's something that we obviously fix.. I'm probably too new to make that call, maybe we need to take it to the community.