Ceph » rgw

thanks for the detailed bug report, I've pushed a PR to s3-tests that adds testing for put_object with and without the ACL, that reproduces this bug https://github.com/ceph/s3-tests/pull/541

the issue here is the computed V4 signature differs from the client vs what the server (radosgw) calculates, this seems to be because of some
internal issue in radosgw where it cannot lookup the HTTP_X_AMZ_ACL header in it's environment (it's empty).

it can be clearly seen by this line generated from here https://github.com/ceph/ceph/blob/4103b566e4b177a2d12932604b1df868022ef5d8/src/rgw/rgw_auth_s3.cc#L717

2024-02-03T12:17:43.664+0000 7f563b62d640 10 warning env var not available HTTP_X_AMZ_ACL

for some reason we cannot read that and thus the signature on the server-side is not calculated correctly including that header. we also store that data
in req_state->canned_acl for a request so i tried a simple hack https://github.com/tobias-urdin/ceph/commit/59126d449f472a8d7973e7c94d34b3de8b51b168 to see
if that was populated, but that also failed.

for for some reason we cannot get HTTP_X_AMZ_ACL header either from req_state->canned_acl or using req_state->info.get("HTTP_X_AMZ_ACL") and i'm a little
bit out of ideas, perhaps somebody else knows why it's not populated?

here is a complete request where the signature fails:

024-02-03T12:17:43.568+0000 7f56bd132640 20 link_request req_data=0x560c3509cb40 req_data->id=7, curl_handle=0x560c34f8f320
2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::keystone::EC2Engine denied with reason=-2028
2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028
2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s v4 credential format = 0555b35654ad1656d804/20240203/us-east-1/s3/aws4_request
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s access key id = 0555b35654ad1656d804
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s credential scope = 20240203/us-east-1/s3/aws4_request
2024-02-03T12:17:43.664+0000 7f563b62d640 10 warning env var not available HTTP_X_AMZ_ACL
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical headers format = host:localhost:8000

2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical req method = PUT, due to access-control-request-method header
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s payload request hash = UNSIGNED-PAYLOAD
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical request = PUT
/yournamehere-h5vili87wkmrue5n-1/foo
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0555b35654ad1656d804%2F20240203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240203T121743Z&X-Amz-Expires=100000&X-Amz-SignedHeaders=host%3Bx-amz-acl
host:localhost:8000

host;x-amz-acl
UNSIGNED-PAYLOAD
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical request hash = e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s string to sign = AWS4-HMAC-SHA256
20240203T121743Z
20240203/us-east-1/s3/aws4_request
e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s date_k    = a309635ea418e8ae49eb17427153f9d1ed675d6b57ef4c98918c255e42e4f929
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s region_k  = da8b65c637e76cecbff714e0a63def4c8cbf3a920a924a5dedd005763596cc0e
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s service_k = ae2f22af6031ce83dc662105dbec695449a0d891442b233f970d2b3e16d26fc9
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s signing_k = fa77f584bd38ffd6bb21cb6257c8213811f398ac9d2124c70b5704b50d883485
2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s generated signature = 820651d5112dabd05dd1eb4a19829550f1450717deb133738bff1bd58c57b1b8
2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors string_to_sign=AWS4-HMAC-SHA256
20240203T121743Z
20240203/us-east-1/s3/aws4_request
e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128
2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors server signature=820651d5112dabd05dd1eb4a19829550f1450717deb133738bff1bd58c57b1b8
2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors client signature=79a130ec2b6befaa84d292dd23878687252cbdfc0f8c67f5ca4328ac22c08686

also – just to clear, it's not strictly a regression but an issue now that we verify the v4 auth (signature) for HTTP OPTIONS calls if provided by the client, but yes it's a bug

wait a minute, the browser cannot possibly know that it should send a x-amz-acl header for the HTTP OPTIONS preflight, so we should just ignore it?

so something like; in get_v4_canonical_headers() we should drop signed headers from calculation if it's a HTTP OPTIONS call and the data could not be found.

so according to this https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html one should use generate_presigned_post() when doing requests that include signed headers so i'm thinking this might be an user error? because we can never validate signed signatures for a HTTP OPTIONS call because they will never be there, see https://stackoverflow.com/questions/52625812/boto3-generate-presigned-url-for-put-object-return-the-request-signature-we

so a user that wants to send x-amz-acl would need to use generate_presigned_post() and not generate_presigned_url() but we should probably verify that and add testing

here is two patches to improve the testing on this area
https://github.com/ceph/s3-tests/pull/542
https://github.com/ceph/s3-tests/pull/543

i created a AWS S3 account and tested this logic

import boto3
from botocore.client import Config
import requests

client = boto3.client(
    's3',
    endpoint_url='https://s3.eu-north-1.amazonaws.com',
    region_name='eu-north-1',
    config=Config(signature_version='s3v4'),
)

params = {
    'Bucket': 'demo-presigned-put',
    'Key': 'test.txt'
}

print('=== GET ===')

url = client.generate_presigned_url(
    ClientMethod='get_object',
    Params=params,
    ExpiresIn=600,
    HttpMethod='GET'
)

print(url)

headers = {
    'Origin': 'example',
    'Access-Control-Request-Method': 'GET',
}

resp = requests.options(url, headers=headers)
print(resp.status_code)
print(resp.text)

print('=== PUT ===')

url = client.generate_presigned_url(
    ClientMethod='put_object',
    Params=params,
    ExpiresIn=600,
    HttpMethod='PUT'
)

print(url)

headers = {
    'Origin': 'example',
    'Access-Control-Request-Method': 'PUT',
}

resp = requests.options(url, headers=headers)
print(resp.status_code)
print(resp.text)

print('=== PUT with ACL in params ===')

params['ACL'] = 'private'

url = client.generate_presigned_url(
    ClientMethod='put_object',
    Params=params,
    ExpiresIn=600,
    HttpMethod='PUT'
)

print(url)

headers = {
    'Origin': 'example',
    'Access-Control-Request-Method': 'PUT',
}

resp = requests.options(url, headers=headers)
print(resp.status_code)
print(resp.text)

the result:

=== GET ===
https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b5eed4a579269bd947577d9c7b4130b5ff4a50e86ed99725b03d3c2216cc8533
200
=== PUT ===
https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=de903d0fb33aa7fa0d297835476c8c20d638825e72532429561deb206e37ecd2
200
=== PUT with ACL in params ===
https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host%3Bx-amz-acl&X-Amz-Signature=9d7677c4186d45722e4d0a08bbe0d9deb779827622e62b3d7a457ec01c359293
200

the conclusion is: this bug is valid and we should allow x-amz-acl in the url for a presigned put_object URL

tests: https://github.com/ceph/s3-tests/pull/544
pr for workaround: https://github.com/ceph/ceph/pull/55458

Will the backports make it into the next release of Quincy/Reef?