Bug #64308
openCORS Preflight Failure After Upgrading to 17.2.7
0%
Description
After upgrading to 17.2.7 we have some users complaining that they can no longer do PUTs with presigned URLs. They are receiving messages like "Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource."
I think it is caused by the change in https://tracker.ceph.com/issues/62033 which is only in 17.2.7 and 18.2.1. 16.2.15 will have it when released.
I was able to reproduce an error in an OPTIONS call to RGW using the attached script.
Create bucket. Apply CORS rules. Set AWS credentials. Run attached script.
In 17.2.7 most of the tests result in 403.
Region us-east-1 Without ACL https://endpoint/bucket/foo.png?AWSAccessKeyId=UFK4WVCRL8XHSMQERIGJ&Signature=tvGiXAca%2B6m8y5YDbDSPY1akqlI%3D&Expires=1706902836 403 With ACL https://endpoint/bucket/foo.png?AWSAccessKeyId=UFK4WVCRL8XHSMQERIGJ&Signature=3pYkXrAajuOFYbTqhq3TY7YcioE%3D&x-amz-acl=private&Expires=1706902837 403 Region us-east-2 Without ACL https://endpoint/bucket/foo.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=UFK4WVCRL8XHSMQERIGJ%2F20240202%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240202T193037Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=a1e723930c116fc45244adf9f2e629b2c5b989480a37eb31b0be38c980dbfc1e 200 With ACL https://endpoint/bucket/foo.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=UFK4WVCRL8XHSMQERIGJ%2F20240202%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240202T193037Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host%3Bx-amz-acl&X-Amz-Signature=cd514a13b7e0327679ec0d3a53e122f8125f3b8c6a35b3851c40b4f0b65058e5 403
In 17.2.5 all of the tests result in 200.
Region us-east-1 Without ACL https://endpoint/bucket/foo.png?AWSAccessKeyId=F5S36GRYN612SREULGN1&Signature=JsLqb4yl%2F3KC8%2B7gcaQ%2BXclHwOA%3D&Expires=1706904230 200 With ACL https://endpoint/bucket/foo.png?AWSAccessKeyId=F5S36GRYN612SREULGN1&Signature=cGAbmXGV0Y29%2BfhLzy4qJdl98XY%3D&x-amz-acl=private&Expires=1706904230 200 Region us-east-2 Without ACL https://endpoint/bucket/foo.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=F5S36GRYN612SREULGN1%2F20240202%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240202T195351Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4376b79a1fabb7747c1022208967c18907240e9d162f2a173508d7152e3effa0 200 With ACL https://endpoint/bucket/foo.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=F5S36GRYN612SREULGN1%2F20240202%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20240202T195352Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host%3Bx-amz-acl&X-Amz-Signature=3aba88be6ceb30d4de3ebaf97529185105156ae005af53aac1e9b377fa6d68ed 200
Files
Updated by Casey Bodley 3 months ago
- Tags set to sigv4 presigned
- Backport set to quincy reef
- Regression changed from No to Yes
Updated by Tobias Urdin 3 months ago
thanks for the detailed bug report, I've pushed a PR to s3-tests that adds testing for put_object with and without the ACL, that reproduces this bug https://github.com/ceph/s3-tests/pull/541
the issue here is the computed V4 signature differs from the client vs what the server (radosgw) calculates, this seems to be because of some
internal issue in radosgw where it cannot lookup the HTTP_X_AMZ_ACL header in it's environment (it's empty).
it can be clearly seen by this line generated from here https://github.com/ceph/ceph/blob/4103b566e4b177a2d12932604b1df868022ef5d8/src/rgw/rgw_auth_s3.cc#L717
2024-02-03T12:17:43.664+0000 7f563b62d640 10 warning env var not available HTTP_X_AMZ_ACL
for some reason we cannot read that and thus the signature on the server-side is not calculated correctly including that header. we also store that data
in req_state->canned_acl for a request so i tried a simple hack https://github.com/tobias-urdin/ceph/commit/59126d449f472a8d7973e7c94d34b3de8b51b168 to see
if that was populated, but that also failed.
for for some reason we cannot get HTTP_X_AMZ_ACL header either from req_state->canned_acl or using req_state->info.get("HTTP_X_AMZ_ACL") and i'm a little
bit out of ideas, perhaps somebody else knows why it's not populated?
here is a complete request where the signature fails:
024-02-03T12:17:43.568+0000 7f56bd132640 20 link_request req_data=0x560c3509cb40 req_data->id=7, curl_handle=0x560c34f8f320 2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::keystone::EC2Engine denied with reason=-2028 2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2024-02-03T12:17:43.664+0000 7f563b62d640 20 req 844319591127462431 0.096010894s s3:options_cors rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s v4 credential format = 0555b35654ad1656d804/20240203/us-east-1/s3/aws4_request 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s access key id = 0555b35654ad1656d804 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s credential scope = 20240203/us-east-1/s3/aws4_request 2024-02-03T12:17:43.664+0000 7f563b62d640 10 warning env var not available HTTP_X_AMZ_ACL 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical headers format = host:localhost:8000 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical req method = PUT, due to access-control-request-method header 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s payload request hash = UNSIGNED-PAYLOAD 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical request = PUT /yournamehere-h5vili87wkmrue5n-1/foo X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=0555b35654ad1656d804%2F20240203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240203T121743Z&X-Amz-Expires=100000&X-Amz-SignedHeaders=host%3Bx-amz-acl host:localhost:8000 host;x-amz-acl UNSIGNED-PAYLOAD 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s canonical request hash = e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s string to sign = AWS4-HMAC-SHA256 20240203T121743Z 20240203/us-east-1/s3/aws4_request e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s date_k = a309635ea418e8ae49eb17427153f9d1ed675d6b57ef4c98918c255e42e4f929 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s region_k = da8b65c637e76cecbff714e0a63def4c8cbf3a920a924a5dedd005763596cc0e 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s service_k = ae2f22af6031ce83dc662105dbec695449a0d891442b233f970d2b3e16d26fc9 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s signing_k = fa77f584bd38ffd6bb21cb6257c8213811f398ac9d2124c70b5704b50d883485 2024-02-03T12:17:43.664+0000 7f563b62d640 10 req 844319591127462431 0.096010894s generated signature = 820651d5112dabd05dd1eb4a19829550f1450717deb133738bff1bd58c57b1b8 2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors string_to_sign=AWS4-HMAC-SHA256 20240203T121743Z 20240203/us-east-1/s3/aws4_request e9d1eda565c99908031258beeb1f49e3c1f2d316502f25f4396900ca928da128 2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors server signature=820651d5112dabd05dd1eb4a19829550f1450717deb133738bff1bd58c57b1b8 2024-02-03T12:17:43.664+0000 7f563b62d640 15 req 844319591127462431 0.096010894s s3:options_cors client signature=79a130ec2b6befaa84d292dd23878687252cbdfc0f8c67f5ca4328ac22c08686
Updated by Tobias Urdin 3 months ago
also – just to clear, it's not strictly a regression but an issue now that we verify the v4 auth (signature) for HTTP OPTIONS calls if provided by the client, but yes it's a bug
Updated by Tobias Urdin 3 months ago
wait a minute, the browser cannot possibly know that it should send a x-amz-acl header for the HTTP OPTIONS preflight, so we should just ignore it?
so something like; in get_v4_canonical_headers() we should drop signed headers from calculation if it's a HTTP OPTIONS call and the data could not be found.
Updated by Tobias Urdin 3 months ago
so according to this https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html one should use generate_presigned_post() when doing requests that include signed headers so i'm thinking this might be an user error? because we can never validate signed signatures for a HTTP OPTIONS call because they will never be there, see https://stackoverflow.com/questions/52625812/boto3-generate-presigned-url-for-put-object-return-the-request-signature-we
so a user that wants to send x-amz-acl would need to use generate_presigned_post() and not generate_presigned_url() but we should probably verify that and add testing
Updated by Tobias Urdin 3 months ago
here is two patches to improve the testing on this area
https://github.com/ceph/s3-tests/pull/542
https://github.com/ceph/s3-tests/pull/543
Updated by Tobias Urdin 3 months ago
i created a AWS S3 account and tested this logic
import boto3 from botocore.client import Config import requests client = boto3.client( 's3', endpoint_url='https://s3.eu-north-1.amazonaws.com', region_name='eu-north-1', config=Config(signature_version='s3v4'), ) params = { 'Bucket': 'demo-presigned-put', 'Key': 'test.txt' } print('=== GET ===') url = client.generate_presigned_url( ClientMethod='get_object', Params=params, ExpiresIn=600, HttpMethod='GET' ) print(url) headers = { 'Origin': 'example', 'Access-Control-Request-Method': 'GET', } resp = requests.options(url, headers=headers) print(resp.status_code) print(resp.text) print('=== PUT ===') url = client.generate_presigned_url( ClientMethod='put_object', Params=params, ExpiresIn=600, HttpMethod='PUT' ) print(url) headers = { 'Origin': 'example', 'Access-Control-Request-Method': 'PUT', } resp = requests.options(url, headers=headers) print(resp.status_code) print(resp.text) print('=== PUT with ACL in params ===') params['ACL'] = 'private' url = client.generate_presigned_url( ClientMethod='put_object', Params=params, ExpiresIn=600, HttpMethod='PUT' ) print(url) headers = { 'Origin': 'example', 'Access-Control-Request-Method': 'PUT', } resp = requests.options(url, headers=headers) print(resp.status_code) print(resp.text)
the result:
=== GET === https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=b5eed4a579269bd947577d9c7b4130b5ff4a50e86ed99725b03d3c2216cc8533 200 === PUT === https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=de903d0fb33aa7fa0d297835476c8c20d638825e72532429561deb206e37ecd2 200 === PUT with ACL in params === https://s3.eu-north-1.amazonaws.com/demo-presigned-put/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXYKJWSKH2WCFPZ66%2F20240205%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240205T093644Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host%3Bx-amz-acl&X-Amz-Signature=9d7677c4186d45722e4d0a08bbe0d9deb779827622e62b3d7a457ec01c359293 200
Updated by Tobias Urdin 3 months ago
the conclusion is: this bug is valid and we should allow x-amz-acl in the url for a presigned put_object URL
Updated by Tobias Urdin 3 months ago
tests: https://github.com/ceph/s3-tests/pull/544
pr for workaround: https://github.com/ceph/ceph/pull/55458
Updated by Casey Bodley 3 months ago
- Status changed from New to Pending Backport
Updated by Backport Bot 3 months ago
- Copied to Backport #64398: reef: CORS Preflight Failure After Upgrading to 17.2.7 added
Updated by Backport Bot 3 months ago
- Copied to Backport #64399: quincy: CORS Preflight Failure After Upgrading to 17.2.7 added
Updated by Backport Bot 3 months ago
- Tags changed from sigv4 presigned to sigv4 presigned backport_processed
Updated by Casey Bodley 3 months ago
- Tags changed from sigv4 presigned backport_processed to sigv4 presigned
- Backport changed from quincy reef to pacific quincy reef
Updated by Backport Bot 3 months ago
- Tags changed from sigv4 presigned to sigv4 presigned backport_processed
Updated by Casey Bodley 3 months ago
- Tags changed from sigv4 presigned backport_processed to sigv4 presigned
Updated by Backport Bot 3 months ago
- Copied to Backport #64404: pacific: CORS Preflight Failure After Upgrading to 17.2.7 added
Updated by Backport Bot 3 months ago
- Tags changed from sigv4 presigned to sigv4 presigned backport_processed
Updated by Reid Guyett 25 days ago
Will the backports make it into the next release of Quincy/Reef?