Project

General

Profile

Bug #46567

Access denied for multi-object-delete by non-bucket-owner

Added by Chris Palmer over 2 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
octopus nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Unexpected 403 access denied response from the following: * Bucket mybucket owned by user "c" * Bucket policy grants s3:listBucket on mybucket, and s3:putObject &
s3:deleteObject on mybucket/* to user "j", and s3:getObject to * (I
even granted s3:* on mybucket/* to "j" with no effect) * User "j" can create objects in mybucket, and can delete individual
objects (using DELETE) * User "j" get 403 when trying to do a multi-object-delete (POST
/mybucket/?delete with a list of 4 object keys)

Code is a Java servlet running in Wildfly, loading its credentials from the default ~/.aws/credentials file, and using the AWS API jar. It enables path-style access. If I change the credentials to those of the bucket owner "c" it works...

Log file shows access has been granted, but further down there is a suspicious "Permissions for user not found" (don't know if that is expected or not).

2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::LocalEngine granted access
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::AWSAuthStrategy granted access
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete normalizing buckets and tenants
2020-07-11T17:55:54.038+0100 7f45adad7700 10 s->object=<NULL> s->bucket=mybucket
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc288 obj=default.rgw.meta:root:mybucket state=0x5628b912e9a0 s->prefetch_data=0
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x16, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x11, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>c</ID><DisplayName>C</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>c</ID><DisplayName>C</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc668 obj=default.rgw.meta:users.uid:j state=0x5628b912e9a0 s->prefetch_data=0
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x6, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty
2020-07-11T17:55:54.038+0100 7f45adad7700 20 Read xattr: user.rgw.idtag
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x3, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete recalculating target
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete reading permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init op
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op mask
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete required_mask= 4 user.op_mask=7
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete -- Getting permissions begin with perm_mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for uid=j
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for user not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=1 mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=2 mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete -
Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0), owner=c, perm=0
2020-07-11T17:55:54.038+0100 7f45adad7700 10 req 15 0.004000002s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=0, user_perm_mask=2, acl perm=0
2020-07-11T17:55:54.038+0100 7f45adad7700 1 op->ERRORHANDLER: err_no=-13 new_err_no=-13
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete op status=0
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete http status=403
2020-07-11T17:55:54.038+0100 7f45adad7700 1 ====== req done req=0x7f45adaced50 op status=0 http_status=403 latency=0.004000002s ======
2020-07-11T17:55:54.038+0100 7f45adad7700 20 process_request() returned -13
2020-07-11T17:55:54.038+0100 7f45adad7700 1 civetweb: 0x5628b9424000: 192.168.80.135 - - [11/Jul/2020:17:55:54 +0100] "POST /mybucket/?delete HTTP/1.1" 403 464 - aws-sdk-java/1.11.820 Linux/5.7.7-200.fc32.x86_64 OpenJDK_64-Bit_Server_VM/14.0.1+7 java/14.0.1 vendor/Red_Hat,_Inc.


Related issues

Copied to rgw - Backport #48547: nautilus: Access denied for multi-object-delete by non-bucket-owner Rejected
Copied to rgw - Backport #48548: octopus: Access denied for multi-object-delete by non-bucket-owner Rejected

History

#1 Updated by Casey Bodley over 2 years ago

can you share your bucket policy?

#2 Updated by Casey Bodley over 2 years ago

  • Status changed from New to Need More Info

#3 Updated by Chris Palmer over 2 years ago

Bucket policy is as below.... Thanks

{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:listBucket" 
                   ],
         "Resource": "arn:aws:s3:::mybucket" 
      },
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:putObject",
                     "s3:deleteObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      },
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": [
                     "s3:getObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      }
   ]
}

#4 Updated by Casey Bodley over 2 years ago

  • Status changed from Need More Info to New

#5 Updated by Casey Bodley over 2 years ago

  • Status changed from New to Triaged

#6 Updated by Casey Bodley over 2 years ago

  • Assignee set to Abhishek Lekshmanan

#7 Updated by Casey Bodley over 2 years ago

  • Priority changed from Normal to High

#8 Updated by Abhishek Lekshmanan over 2 years ago

  • Status changed from Triaged to In Progress
  • Pull request ID set to 36583

#9 Updated by Abhishek Lekshmanan over 2 years ago

couldn't reproduce in octopus/nautilus, could find a problem in master.

#10 Updated by Chris Palmer over 2 years ago

I just tried reproducing it in Octopus 15.2.4.
Here's the result which shows the problem.

radosgw-admin user create --uid=c --display-name=C
    "keys": [
        {
            "user": "c",
            "access_key": "83YNHZ7V5FSAA9T1BCED",
            "secret_key": "aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo" 
        }
    ],

radosgw-admin user create --uid=j --display-name=J
    "keys": [
        {
            "user": "j",
            "access_key": "PWYGUSSAU2N0WJTJTG3Y",
            "secret_key": "INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY" 
        }
    ],

cat <<EOF >> ~/.aws/credentials
[c]
aws_access_key_id = 83YNHZ7V5FSAA9T1BCED
aws_secret_access_key = aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo

[j]
aws_access_key_id = PWYGUSSAU2N0WJTJTG3Y
aws_secret_access_key = INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY
EOF

cat <<EOF > mybucket-policy.json
{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:listBucket" 
                   ],
         "Resource": "arn:aws:s3:::mybucket" 
      },
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:putObject",
                     "s3:deleteObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      },
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": [
                     "s3:getObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      }
   ]
}
EOF

export S3="aws --endpoint-url http://xxx.yyy:80 s3" 
export S3API="aws --endpoint-url http://xxx.yyy:80 s3api" 

$S3 --profile c mb s3://mybucket
make_bucket: mybucket

$S3API --profile c put-bucket-policy --bucket mybucket --policy file://mybucket-policy.json

$S3 --profile j cp /etc/hosts s3://mybucket/hosts
upload: ../../etc/hosts to s3://mybucket/hosts

###### This confirms j can delete files
$S3 --profile j rm s3://mybucket/hosts
delete: s3://mybucket/hosts

$S3 --profile j cp /etc/hosts s3://mybucket/hosts
upload: ../../etc/hosts to s3://mybucket/hosts

###### This error is unexpected
$S3API --profile j delete-objects --bucket mybucket --delete Objects=[{Key=hosts}]
An error occurred (AccessDenied) when calling the DeleteObjects operation: Unknown

###### This confirms the above works with owner credentials
$S3API --profile c delete-objects --bucket mybucket --delete Objects=[{Key=hosts}]
{
    "Deleted": [
        {
            "Key": "hosts" 
        }
    ]
}

#11 Updated by Casey Bodley about 2 years ago

  • Status changed from In Progress to Fix Under Review

#13 Updated by Abhishek Lekshmanan about 2 years ago

  • Pull request ID changed from 36583 to 37933

#14 Updated by Casey Bodley almost 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to octopus

#15 Updated by Casey Bodley almost 2 years ago

  • Backport changed from octopus to octopus nautilus

#16 Updated by Backport Bot almost 2 years ago

  • Copied to Backport #48547: nautilus: Access denied for multi-object-delete by non-bucket-owner added

#17 Updated by Backport Bot almost 2 years ago

  • Copied to Backport #48548: octopus: Access denied for multi-object-delete by non-bucket-owner added

#18 Updated by Chris Palmer over 1 year ago

This has been pending backport for a very long time now. We've since upgraded to Pacific 16.2.5 where it is still broken. Any chance of this being progressed?

Thanks

#19 Updated by JS Landry about 1 year ago

Hi, it's look like I hit this bug(?) too.

In my case, I test 3 clients: s3browser, cyberduck and s3cmd, and the error is present only with s3browser. -- https://s3browser.com/

I'm not sure where the problem is, but s3browser, when doing "bulk delete" (ie. delete more than one object) send a single POST -- as described here: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

but when using cyberduck and s3cmd, these clients send many DELETE events.

s3browser single object delete works, using a single DELETE event. Here's the syslog msg on the radosgw server:

rgw05 haproxy[123175]: 192.168.209.74:58386 [01/Dec/2021:16:19:16.422] s3~ radosgw/rgw05 0/0/0/9/9 204 156 - - ---- 4/3/0/0/0 0/0 "DELETE /testbucket/test321/test-js1.dat HTTP/1.1" 

and the s3browser "bulk objects delete", doesn't work, I got a "Failed - AccessDenied, 403 Forbidden" in s3browser, and this event in the radosgw syslog:

rgw08 haproxy[13544]: 192.168.209.74:43546 [01/Dec/2021:16:20:53.787] s3~ radosgw/rgw08 0/0/1/44/45 403 458 - - ---- 4/3/0/0/0 0/0 "POST /testbucket/?delete= HTTP/1.1" 

using s3cmd to "wildcard delete" works. (and with cyberduck too)

$ s3cmd del s3://testbucket/test321/test-js*
delete: 's3://testbucket/test321/test-js2.dat'
delete: 's3://testbucket/test321/test-js3.dat'

and the syslog for the s3cmd delete:

rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.013] s3~ radosgw/rgw05 0/0/0/13/13 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js2.dat HTTP/1.1" 
rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.078] s3~ radosgw/rgw05 0/0/0/14/14 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js3.dat HTTP/1.1" 

The user is not the bucket owner.
ceph version 15.2.14

Hopping that could help finding something...

#20 Updated by JS Landry about 1 year ago

sorry I may have reply too quickly to this ticket, I just check the https://github.com/ceph/ceph/pull/37933 and it may not related.

#21 Updated by Backport Bot 4 months ago

  • Tags set to backport_processed

#22 Updated by Konstantin Shalygin 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF