Ceph » rgw

can you share your bucket policy?

Bucket policy is as below.... Thanks

{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:listBucket" 
                   ],
         "Resource": "arn:aws:s3:::mybucket" 
      },
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:putObject",
                     "s3:deleteObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      },
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": [
                     "s3:getObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      }
   ]
}

couldn't reproduce in octopus/nautilus, could find a problem in master.

I just tried reproducing it in Octopus 15.2.4.
Here's the result which shows the problem.

radosgw-admin user create --uid=c --display-name=C
    "keys": [
        {
            "user": "c",
            "access_key": "83YNHZ7V5FSAA9T1BCED",
            "secret_key": "aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo" 
        }
    ],

radosgw-admin user create --uid=j --display-name=J
    "keys": [
        {
            "user": "j",
            "access_key": "PWYGUSSAU2N0WJTJTG3Y",
            "secret_key": "INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY" 
        }
    ],

cat <<EOF >> ~/.aws/credentials
[c]
aws_access_key_id = 83YNHZ7V5FSAA9T1BCED
aws_secret_access_key = aMVwXRp3YEDAGmESMCs1KIYF0xMZcWaRoBI7AOuo

[j]
aws_access_key_id = PWYGUSSAU2N0WJTJTG3Y
aws_secret_access_key = INHkCyxHfUfnsMtFH2Rxr75Z9MQQtagzcJHddJEY
EOF

cat <<EOF > mybucket-policy.json
{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:listBucket" 
                   ],
         "Resource": "arn:aws:s3:::mybucket" 
      },
      {
         "Effect": "Allow",
         "Principal": {"AWS": [ "arn:aws:iam:::user/j" ]},
         "Action": [
                     "s3:putObject",
                     "s3:deleteObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      },
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": [
                     "s3:getObject" 
                   ],
         "Resource": "arn:aws:s3:::mybucket/*" 
      }
   ]
}
EOF

export S3="aws --endpoint-url http://xxx.yyy:80 s3" 
export S3API="aws --endpoint-url http://xxx.yyy:80 s3api" 

$S3 --profile c mb s3://mybucket
make_bucket: mybucket

$S3API --profile c put-bucket-policy --bucket mybucket --policy file://mybucket-policy.json

$S3 --profile j cp /etc/hosts s3://mybucket/hosts
upload: ../../etc/hosts to s3://mybucket/hosts

###### This confirms j can delete files
$S3 --profile j rm s3://mybucket/hosts
delete: s3://mybucket/hosts

$S3 --profile j cp /etc/hosts s3://mybucket/hosts
upload: ../../etc/hosts to s3://mybucket/hosts

###### This error is unexpected
$S3API --profile j delete-objects --bucket mybucket --delete Objects=[{Key=hosts}]
An error occurred (AccessDenied) when calling the DeleteObjects operation: Unknown

###### This confirms the above works with owner credentials
$S3API --profile c delete-objects --bucket mybucket --delete Objects=[{Key=hosts}]
{
    "Deleted": [
        {
            "Key": "hosts" 
        }
    ]
}

included in https://github.com/ceph/ceph/pull/37933

This has been pending backport for a very long time now. We've since upgraded to Pacific 16.2.5 where it is still broken. Any chance of this being progressed?

Thanks

Hi, it's look like I hit this bug(?) too.

In my case, I test 3 clients: s3browser, cyberduck and s3cmd, and the error is present only with s3browser. -- https://s3browser.com/

I'm not sure where the problem is, but s3browser, when doing "bulk delete" (ie. delete more than one object) send a single POST -- as described here: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

but when using cyberduck and s3cmd, these clients send many DELETE events.

s3browser single object delete works, using a single DELETE event. Here's the syslog msg on the radosgw server:

rgw05 haproxy[123175]: 192.168.209.74:58386 [01/Dec/2021:16:19:16.422] s3~ radosgw/rgw05 0/0/0/9/9 204 156 - - ---- 4/3/0/0/0 0/0 "DELETE /testbucket/test321/test-js1.dat HTTP/1.1" 

and the s3browser "bulk objects delete", doesn't work, I got a "Failed - AccessDenied, 403 Forbidden" in s3browser, and this event in the radosgw syslog:

rgw08 haproxy[13544]: 192.168.209.74:43546 [01/Dec/2021:16:20:53.787] s3~ radosgw/rgw08 0/0/1/44/45 403 458 - - ---- 4/3/0/0/0 0/0 "POST /testbucket/?delete= HTTP/1.1" 

using s3cmd to "wildcard delete" works. (and with cyberduck too)

$ s3cmd del s3://testbucket/test321/test-js*
delete: 's3://testbucket/test321/test-js2.dat'
delete: 's3://testbucket/test321/test-js3.dat'

and the syslog for the s3cmd delete:

rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.013] s3~ radosgw/rgw05 0/0/0/13/13 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js2.dat HTTP/1.1" 
rgw05 haproxy[123175]: 192.168.87.160:37190 [01/Dec/2021:16:24:06.078] s3~ radosgw/rgw05 0/0/0/14/14 204 156 - - ---- 8/7/0/0/0 0/0 "DELETE /test321/test-js3.dat HTTP/1.1" 

The user is not the bucket owner.
ceph version 15.2.14

Hopping that could help finding something...

sorry I may have reply too quickly to this ticket, I just check the https://github.com/ceph/ceph/pull/37933 and it may not related.

Hello! Was this actually fixed, or was it marked “Resolved” for another reason? I can replicate the issue using Rook v1.10.7 (Ceph v17.2.5 Quincy) with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts" 
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::${S3_BUCKET}" 
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam:::user/${S3_USERNAME}" 
        ]
      }
    },
    {
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl" 
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::${S3_BUCKET}/*" 
      ],
      "Principal": {
        "AWS": [
          "arn:aws:iam:::user/${S3_USERNAME}" 
        ]
      }
    }
  ]
}

I've just checked this in 17.2.5, using my reproducer already given above, and the original problem is still there.

This is also reproducible on 16.2.7 using AWS Java client and test case provided by Chris.

Created new issue for this since this one is marked as Resolved.
Link https://tracker.ceph.com/issues/59474