Ceph

Ceph Versions: 15.2.17 and 17.2.6

We are using STS tokens for temporary S3 access for our clients, using the assume role function of boto3. After updating to 17.2.6 this functionality no longer works, it is possible to successfully generate the token by assuming role to a user that own that bucket but when trying to upload/download files from that bucket a forbidden response is given. I have pasted the error, boto3 code being used, and the RGW logs when debug_rgw is set to 20. I have added a file with the complete rgw debug log.

I have also deployed a new cluster on 15.2.17 that has the same error occur.

Error Log:
Traceback (most recent call last):
File "get_sts.py", line 66, in <module>
s3_client.download_file(bucket_name, file_key, local_file_path)
File "/usr/lib/python3/dist-packages/boto3/s3/inject.py", line 170, in download_file
return transfer.download_file(
File "/usr/lib/python3/dist-packages/boto3/s3/transfer.py", line 307, in download_file
future.result()
File "/usr/lib/python3/dist-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/usr/lib/python3/dist-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 255, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 340, in _submit
response = client.head_object(
File "/usr/lib/python3/dist-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3/dist-packages/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Boto3 Code:
@import boto3
import json
from botocore.config import Config

sts_client = boto3.client(
'sts',
#User: Test
#aws_access_key_id="NDH8W54VZEZG42E8ENOI",
#aws_secret_access_key="kCD9LibehnZZl5LhCXeHb3Mj4NzUhMRHGAxO4Y14",
#User: Austin
aws_access_key_id="MHOJQR0HSXJQZHSFUKAQ",
aws_secret_access_key="5HCLGQUBGJyQJFCCluCHl2GhSvoPZVrrjTCJz6wA",
endpoint_url="http://192.168.175.200:8080",
region_name="",
config=Config(
signature_version='v4-query',
)
)

role_arn = 'arn:aws:iam:::role/DefaultNoS3Access'

response = sts_client.assume_role(
RoleArn=role_arn,
RoleSessionName="test",
Policy=json.dumps( {
"Version": "2012-10-17",
"Statement": [ {
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::active-data",
#"arn:aws:s3:::active-data/*",
"arn:aws:s3:::cold-data/*"
]
}
]
}, separators=(',', ':')
)
)

print(response)

credentials = response['Credentials']

s3_client = boto3.client(
's3',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
endpoint_url="http://192.168.175.200:8080",
region_name=""
)

bucket_name = 'active-data'
file_key = 'test.txt'
local_file_path = '/'

#s3_client.list_buckets()

1. Download the file from S3
   s3_client.download_file(bucket_name, file_key, local_file_path)@

RGW Debug Logs:
@2023-07-03T12:41:54.763+0000 7f5f5566a700 1 ====== starting new request req=0x7f602300b6f0 =====
2023-07-03T12:41:54.763+0000 7f5f5566a700 2 req 17911858806429292722 0.000000000s initializing for trans_id = tx00000f893b4aef02b98b2-0064a2c212-ac45-default
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s rgw api priority: s3=8 s3website=7
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s host=192.168.175.200
2023-07-03T12:41:54.763+0000 7f5f5566a700 20 req 17911858806429292722 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2023-07-03T12:41:54.763+0000 7f5f5566a700 20 req 17911858806429292722 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Action val: AssumeRole
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Version val: 2011-06-15
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: RoleArn val: arn:aws:iam:::role/DefaultNoS3Access
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: RoleSessionName val: test
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Policy val: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObjectAcl","s3:PutObject","s3:ListBucket"],"Resource":["arn:aws:s3:::active-data","arn:aws:s3:::cold-data/*"]}]}
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-algorithm val: AWS4-HMAC-SHA256
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-credential val: MHOJQR0HSXJQZHSFUKAQ/20230703//sts/aws4_request
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-date val: 20230703T124154Z
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-expires val: 3600
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-signedheaders val: host
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-signature val: 1aef3c24dd0e1075d82f779e12d415aa2c87d11ad32eaa37f26fd2266ec76de8
2023-07-03T12:41:54.763+0000 7f5f5566a700 20 req 17911858806429292722 0.000000000s get_handler handler=26RGWHandler_REST_Service_S3
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s handler=26RGWHandler_REST_Service_S3
2023-07-03T12:41:54.763+0000 7f5f5566a700 2 req 17911858806429292722 0.000000000s getting op 4
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Action val: AssumeRole
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Version val: 2011-06-15
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: RoleArn val: arn:aws:iam:::role/DefaultNoS3Access
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: RoleSessionName val: test
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: Policy val: {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:GetObject","s3:PutObjectAcl","s3:PutObject","s3:ListBucket"],"Resource":["arn:aws:s3:::active-data","arn:aws:s3:::cold-data/*"]}]}
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-algorithm val: AWS4-HMAC-SHA256
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-credential val: MHOJQR0HSXJQZHSFUKAQ/20230703//sts/aws4_request
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-date val: 20230703T124154Z
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-expires val: 3600
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-signedheaders val: host
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s name: x-amz-signature val: 1aef3c24dd0e1075d82f779e12d415aa2c87d11ad32eaa37f26fd2266ec76de8
2023-07-03T12:41:54.763+0000 7f5f5566a700 20 req 17911858806429292722 0.000000000s get_system_obj_state: rctx=0x7f602300a730 obj=default.rgw.log:script.prerequest. state=0x7f5fe00251d0 s->prefetch_data=0
2023-07-03T12:41:54.763+0000 7f5f5566a700 10 req 17911858806429292722 0.000000000s cache get: name=default.rgw.log++script.prerequest. : expiry miss
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s cache put: name=default.rgw.log++script.prerequest. info.flags=0x0
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s adding default.rgw.log++script.prerequest. to cache LRU end
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s sts:assume_role scheduling with throttler client=0 cost=1
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s sts:assume_role op=16RGWSTSAssumeRole
2023-07-03T12:41:54.763+0000 7f5f4e65c700 2 req 17911858806429292722 0.000000000s sts:assume_role verifying requester
2023-07-03T12:41:54.763+0000 7f5f4e65c700 20 req 17911858806429292722 0.000000000s sts:assume_role rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2023-07-03T12:41:54.763+0000 7f5f4e65c700 20 req 17911858806429292722 0.000000000s sts:assume_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2023-07-03T12:41:54.763+0000 7f5f4e65c700 20 req 17911858806429292722 0.000000000s sts:assume_role rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2023-07-03T12:41:54.763+0000 7f5f4e65c700 20 req 17911858806429292722 0.000000000s sts:assume_role rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::STSAuthStrategy
2023-07-03T12:41:54.763+0000 7f5f4e65c700 20 req 17911858806429292722 0.000000000s sts:assume_role rgw::auth::s3::STSAuthStrategy: trying rgw::auth::s3::STSEngine
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s v4 credential format = MHOJQR0HSXJQZHSFUKAQ/20230703//sts/aws4_request
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s access key id = MHOJQR0HSXJQZHSFUKAQ
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s credential scope = 20230703//sts/aws4_request
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s canonical headers format = host:192.168.175.200:8080

2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-07-03T12:41:54.763+0000 7f5f4e65c700 10 req 17911858806429292722 0.000000000s canonical request = POST@