Bug #46567
closedAccess denied for multi-object-delete by non-bucket-owner
0%
Description
Unexpected 403 access denied response from the following:
* Bucket mybucket owned by user "c"
* Bucket policy grants s3:listBucket on mybucket, and s3:putObject &
s3:deleteObject on mybucket/* to user "j", and s3:getObject to * (I
even granted s3:* on mybucket/* to "j" with no effect)
* User "j" can create objects in mybucket, and can delete individual
objects (using DELETE)
* User "j" get 403 when trying to do a multi-object-delete (POST
/mybucket/?delete with a list of 4 object keys)
Code is a Java servlet running in Wildfly, loading its credentials from the default ~/.aws/credentials file, and using the AWS API jar. It enables path-style access. If I change the credentials to those of the bucket owner "c" it works...
Log file shows access has been granted, but further down there is a suspicious "Permissions for user not found" (don't know if that is expected or not).
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::LocalEngine granted access
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete rgw::auth::s3::AWSAuthStrategy granted access
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete normalizing buckets and tenants
2020-07-11T17:55:54.038+0100 7f45adad7700 10 s->object=<NULL> s->bucket=mybucket
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc288 obj=default.rgw.meta:root:mybucket state=0x5628b912e9a0 s->prefetch_data=0
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x16, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+root+mybucket : hit (requested=0x11, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>c</ID><DisplayName>C</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>c</ID><DisplayName>C</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: rctx=0x7f45adacc668 obj=default.rgw.meta:users.uid:j state=0x5628b912e9a0 s->prefetch_data=0
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x6, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 20 get_system_obj_state: s->obj_tag was set empty
2020-07-11T17:55:54.038+0100 7f45adad7700 20 Read xattr: user.rgw.idtag
2020-07-11T17:55:54.038+0100 7f45adad7700 10 cache get: name=default.rgw.meta+users.uid+j : hit (requested=0x3, cached=0x17)
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete recalculating target
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete reading permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete init op
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op mask
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete required_mask= 4 user.op_mask=7
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete verifying op permissions
2020-07-11T17:55:54.038+0100 7f45adad7700 20 req 15 0.004000002s s3:multi_object_delete -- Getting permissions begin with perm_mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete Searching permissions for identity=rgw::auth::SysReqApplier > rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) mask=50 Getting permissions done for identity=rgw::auth::SysReqApplier
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for uid=j
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for user not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=1 mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Searching permissions for group=2 mask=50
2020-07-11T17:55:54.038+0100 7f45adad7700 5 Permissions for group not found
2020-07-11T17:55:54.038+0100 7f45adad7700 5 req 15 0.004000002s s3:multi_object_delete -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0), owner=c, perm=0>ERRORHANDLER: err_no=-13 new_err_no=-13
2020-07-11T17:55:54.038+0100 7f45adad7700 10 req 15 0.004000002s s3:multi_object_delete identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=j, acct_name=J, subuser=, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=0, user_perm_mask=2, acl perm=0
2020-07-11T17:55:54.038+0100 7f45adad7700 1 op
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete op status=0
2020-07-11T17:55:54.038+0100 7f45adad7700 2 req 15 0.004000002s s3:multi_object_delete http status=403
2020-07-11T17:55:54.038+0100 7f45adad7700 1 ====== req done req=0x7f45adaced50 op status=0 http_status=403 latency=0.004000002s ======
2020-07-11T17:55:54.038+0100 7f45adad7700 20 process_request() returned -13
2020-07-11T17:55:54.038+0100 7f45adad7700 1 civetweb: 0x5628b9424000: 192.168.80.135 - - [11/Jul/2020:17:55:54 +0100] "POST /mybucket/?delete HTTP/1.1" 403 464 - aws-sdk-java/1.11.820 Linux/5.7.7-200.fc32.x86_64 OpenJDK_64-Bit_Server_VM/14.0.1+7 java/14.0.1 vendor/Red_Hat,_Inc.
Updated by Casey Bodley almost 4 years ago
- Status changed from New to Need More Info
Updated by Chris Palmer almost 4 years ago
Bucket policy is as below.... Thanks
{ "Statement": [ { "Effect": "Allow", "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]}, "Action": [ "s3:listBucket" ], "Resource": "arn:aws:s3:::mybucket" }, { "Effect": "Allow", "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]}, "Action": [ "s3:putObject", "s3:deleteObject" ], "Resource": "arn:aws:s3:::mybucket/*" }, { "Effect": "Allow", "Principal": "*", "Action": [ "s3:getObject" ], "Resource": "arn:aws:s3:::mybucket/*" } ] }
Updated by Casey Bodley almost 4 years ago
- Status changed from Need More Info to New
Updated by Abhishek Lekshmanan over 3 years ago
- Status changed from Triaged to In Progress
- Pull request ID set to 36583
Updated by Abhishek Lekshmanan over 3 years ago
couldn't reproduce in octopus/nautilus, could find a problem in master.
Updated by Chris Palmer over 3 years ago
I just tried reproducing it in Octopus 15.2.4.
Here's the result which shows the problem.
radosgw-admin user create --uid=c --display-name=C "keys": [ { "user": "c", "access_key": "83YNHZ7V5FSAA9T1BCED", "secret_key": "aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo" } ], radosgw-admin user create --uid=j --display-name=J "keys": [ { "user": "j", "access_key": "PWYGUSSAU2N0WJTJTG3Y", "secret_key": "INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY" } ], cat <<EOF >> ~/.aws/credentials [c] aws_access_key_id = 83YNHZ7V5FSAA9T1BCED aws_secret_access_key = aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo [j] aws_access_key_id = PWYGUSSAU2N0WJTJTG3Y aws_secret_access_key = INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY EOF cat <<EOF > mybucket-policy.json { "Statement": [ { "Effect": "Allow", "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]}, "Action": [ "s3:listBucket" ], "Resource": "arn:aws:s3:::mybucket" }, { "Effect": "Allow", "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]}, "Action": [ "s3:putObject", "s3:deleteObject" ], "Resource": "arn:aws:s3:::mybucket/*" }, { "Effect": "Allow", "Principal": "*", "Action": [ "s3:getObject" ], "Resource": "arn:aws:s3:::mybucket/*" } ] } EOF export S3="aws --endpoint-url http://xxx.yyy:80 s3" export S3API="aws --endpoint-url http://xxx.yyy:80 s3api" $S3 --profile c mb s3://mybucket make_bucket: mybucket $S3API --profile c put-bucket-policy --bucket mybucket --policy file://mybucket-policy.json $S3 --profile j cp /etc/hosts s3://mybucket/hosts upload: ../../etc/hosts to s3://mybucket/hosts ###### This confirms j can delete files $S3 --profile j rm s3://mybucket/hosts delete: s3://mybucket/hosts $S3 --profile j cp /etc/hosts s3://mybucket/hosts upload: ../../etc/hosts to s3://mybucket/hosts ###### This error is unexpected $S3API --profile j delete-objects --bucket mybucket --delete Objects=[{Key=hosts}] An error occurred (AccessDenied) when calling the DeleteObjects operation: Unknown ###### This confirms the above works with owner credentials $S3API --profile c delete-objects --bucket mybucket --delete Objects=[{Key=hosts}] { "Deleted": [ { "Key": "hosts" } ] }
Updated by Casey Bodley over 3 years ago
- Status changed from In Progress to Fix Under Review
Updated by Abhishek Lekshmanan over 3 years ago
included in https://github.com/ceph/ceph/pull/37933
Updated by Abhishek Lekshmanan over 3 years ago
- Pull request ID changed from 36583 to 37933
Updated by Casey Bodley over 3 years ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to octopus
Updated by Casey Bodley over 3 years ago
- Backport changed from octopus to octopus nautilus
Updated by Backport Bot over 3 years ago
- Copied to Backport #48547: nautilus: Access denied for multi-object-delete by non-bucket-owner added
Updated by Backport Bot over 3 years ago
- Copied to Backport #48548: octopus: Access denied for multi-object-delete by non-bucket-owner added
Updated by Chris Palmer almost 3 years ago
This has been pending backport for a very long time now. We've since upgraded to Pacific 16.2.5 where it is still broken. Any chance of this being progressed?
Thanks
Updated by JS Landry over 2 years ago
Hi, it's look like I hit this bug(?) too.
In my case, I test 3 clients: s3browser, cyberduck and s3cmd, and the error is present only with s3browser. -- https://s3browser.com/
I'm not sure where the problem is, but s3browser, when doing "bulk delete" (ie. delete more than one object) send a single POST -- as described here: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
but when using cyberduck and s3cmd, these clients send many DELETE events.
s3browser single object delete works, using a single DELETE event. Here's the syslog msg on the radosgw server:
rgw05 haproxy[123175]: 192.168.209.74:58386 [01/Dec/2021:16:19:16.422] s3~ radosgw/rgw05 0/0/0/9/9 204 156 - - ---- 4/3/0/0/0 0/0 "DELETE /testbucket/test321/test-js1.dat HTTP/1.1"
and the s3browser "bulk objects delete", doesn't work, I got a "Failed - AccessDenied, 403 Forbidden" in s3browser, and this event in the radosgw syslog:
rgw08 haproxy[13544]: 192.168.209.74:43546 [01/Dec/2021:16:20:53.787] s3~ radosgw/rgw08 0/0/1/44/45 403 458 - - ---- 4/3/0/0/0 0/0 "POST /testbucket/?delete= HTTP/1.1"
using s3cmd to "wildcard delete" works. (and with cyberduck too)
$ s3cmd del s3://testbucket/test321/test-js* delete: 's3://testbucket/test321/test-js2.dat' delete: 's3://testbucket/test321/test-js3.dat'
and the syslog for the s3cmd delete:
rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.013] s3~ radosgw/rgw05 0/0/0/13/13 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js2.dat HTTP/1.1" rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.078] s3~ radosgw/rgw05 0/0/0/14/14 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js3.dat HTTP/1.1"
The user is not the bucket owner.
ceph version 15.2.14
Hopping that could help finding something...
Updated by JS Landry over 2 years ago
sorry I may have reply too quickly to this ticket, I just check the https://github.com/ceph/ceph/pull/37933 and it may not related.
Updated by Konstantin Shalygin over 1 year ago
- Status changed from Pending Backport to Resolved
Updated by Ezra Celli about 1 year ago
Hello! Was this actually fixed, or was it marked “Resolved” for another reason? I can replicate the issue using Rook v1.10.7 (Ceph v17.2.5 Quincy) with the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::${S3_BUCKET}"
],
"Principal": {
"AWS": [
"arn:aws:iam:::user/${S3_USERNAME}"
]
}
},
{
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::${S3_BUCKET}/*"
],
"Principal": {
"AWS": [
"arn:aws:iam:::user/${S3_USERNAME}"
]
}
}
]
}
Updated by Chris Palmer about 1 year ago
I've just checked this in 17.2.5, using my reproducer already given above, and the original problem is still there.
Updated by Daniel Iwan about 1 year ago
This is also reproducible on 16.2.7 using AWS Java client and test case provided by Chris.
Updated by Daniel Iwan about 1 year ago
Created new issue for this since this one is marked as Resolved.
Link https://tracker.ceph.com/issues/59474
Updated by Casey Bodley about 1 year ago
- Related to Bug #59474: Cannot delete object using multi-delete operation on a bucket with policy added