Bug #44804
closedSigned Url generated with metadata alway return SignatureNotMatch
0%
Description
Hi all.
1. Upload without metadata (successfully)
I use my code to generate url
s3client = boto3.client(service_name='s3', aws_access_key_id=access_key, aws_secret_access_key=secret_key, endpoint_url=endpoint_url, use_ssl=False, verify=False, config=Config(signature_version='s3v4')) bucket_name = 'bucket3' s3client.create_bucket(Bucket=bucket_name) url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg'}, ExpiresIn=3600, HttpMethod='PUT') print(url)
I have presigned url to put object
http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b
Then i use curl
curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b"
I upload file successfully.
2. Upload with metadata (403 SignatureNotMatch)
But I add more header metadata
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg','ContentType':'image/jpeg','ACL':'public-read'}, ExpiresIn=3600, HttpMethod='PUT') print(url)
Request return 403 SignatureNotMatch
This is log debug boto3
2020-03-30 14:44:05,027 botocore.auth [DEBUG] Calculating signature using v4 auth. 2020-03-30 14:44:05,028 botocore.auth [DEBUG] CanonicalRequest: PUT /bucket3/1.jpg X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl content-type:image/jpeg host:10.5.8.156 x-amz-acl:public-read content-type;host;x-amz-acl UNSIGNED-PAYLOAD 2020-03-30 14:44:05,028 botocore.auth [DEBUG] StringToSign: AWS4-HMAC-SHA256 20200330T074405Z 20200330/us-east-1/s3/aws4_request c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391 2020-03-30 14:44:05,029 botocore.auth [DEBUG] Signature: 4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f
Then i use curl command to upload
curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f" It rerturn : <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000038-005e81a380-2122-default</RequestId><HostId>2122-default-default</HostId>
This is rgw debug log.
2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical headers format = host:10.5.8.156 2020-03-30 14:45:04.323 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD 2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request = PUT /bucket3/1.jpg X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl host:10.5.8.156 content-type;host;x-amz-acl UNSIGNED-PAYLOAD 2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request hash = 15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22 2020-03-30 14:45:04.323 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256 20200330T074405Z 20200330/us-east-1/s3/aws4_request 15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22 2020-03-30 14:45:04.323 7f4bdbcf0700 10 date_k = b24064027f23cd98c3255b42b2ee511d907663f0e29c42e15e8f2b843a231cbc 2020-03-30 14:45:04.323 7f4bdbcf0700 10 region_k = 4a1865641ecc0ab67a6dced8b6d09822752c873b8a5d19bacb6139f10a01e7dd 2020-03-30 14:45:04.323 7f4bdbcf0700 10 service_k = a2eff4dc70d9f170de38439e4732287358c3ebd821d9d354c188ca409a09fbbb 2020-03-30 14:45:04.323 7f4bdbcf0700 10 signing_k = 6be6df1acb55821c2571af97a21845caf277ce8007103d7bde1188c5b0403528 2020-03-30 14:45:04.323 7f4bdbcf0700 10 generated signature = 6ad1438a1eded893f0f89153efeaf00d97129d99bd4c2f9467096704f8cd7e57 2020-03-30 14:45:04.323 7f4bdbcf0700 5 Failed the auth strategy, reason=-2027 2020-03-30 14:45:04.323 7f4bdbcf0700 10 failed to authorize request 2020-03-30 14:45:04.327 7f4bdbcf0700 2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:op status=0 2020-03-30 14:45:04.327 7f4bdbcf0700 2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:http status=403
3. Upload with metadata successfully (add more header in curl command to work arround)
curl --request PUT --upload-file 1.jpg -H 'Content-Type:image/jpeg' -H 'x-amz-acl:public-read' "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f"
This is rgw log
2020-03-30 14:50:09.071 7f4bdbcf0700 10 credential scope = 20200330/us-east-1/s3/aws4_request 2020-03-30 14:50:09.071 7f4bdbcf0700 10 canonical headers format = content-type:image/jpeg host:10.5.8.156 x-amz-acl:public-read 2020-03-30 14:50:09.071 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD 2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request = PUT /bucket3/1.jpg X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl content-type:image/jpeg host:10.5.8.156 x-amz-acl:public-read content-type;host;x-amz-acl UNSIGNED-PAYLOAD 2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request hash = c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391 2020-03-30 14:50:09.075 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256 20200330T074405Z 20200330/us-east-1/s3/aws4_request c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391
Rgw not add metadata from url in
http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09flink
to canonical request.
I think rgw need add parameter from query string in signed url to canonical request.
Thanks.
Updated by Or Friedmann about 4 years ago
Hi Hoan,
I ran this script:
import boto3
import botocore
s3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)
then run:
curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"
It works
Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)
Updated by hoan nv about 4 years ago
Or Friedmann wrote:
Hi Hoan,
I ran this script:
import boto3
import botocores3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)then run:
curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"
It works
Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)
I was tested this script with Amazon S3, it wwork. Curl command don't need add content-type and x-amz-acl headers to curl request. I think rgw should work same as Amazon S3.
Updated by Or Friedmann about 4 years ago
curl --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
curl -H "Content-Type: image/jpeg" -H "x-amz-acl: private" --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"
worked
It looks like it is mandatory in AWS too to send the signed headers.