Project

General

Profile

Bug #44804

Signed Url generated with metadata alway return SignatureNotMatch

Added by hoan nv 3 months ago. Updated 2 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature:

Description

Hi all.

1. Upload without metadata (successfully)
I use my code to generate url

s3client = boto3.client(service_name='s3',
                  aws_access_key_id=access_key,
                  aws_secret_access_key=secret_key,
                  endpoint_url=endpoint_url, use_ssl=False, verify=False,
                  config=Config(signature_version='s3v4'))
bucket_name = 'bucket3'
s3client.create_bucket(Bucket=bucket_name)
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

I have presigned url to put object

http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b

Then i use curl

curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b" 

I upload file successfully.

2. Upload with metadata (403 SignatureNotMatch)

But I add more header metadata

url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg','ContentType':'image/jpeg','ACL':'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

Request return 403 SignatureNotMatch

This is log debug boto3

2020-03-30 14:44:05,027 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-03-30 14:44:05,028 botocore.auth [DEBUG] CanonicalRequest:
PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:44:05,028 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391
2020-03-30 14:44:05,029 botocore.auth [DEBUG] Signature:
4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f
http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f

Then i use curl command to upload

curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f" 

It rerturn :
<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000038-005e81a380-2122-default</RequestId><HostId>2122-default-default</HostId>

This is rgw debug log.

2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical headers format = host:10.5.8.156

2020-03-30 14:45:04.323 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD
2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request = PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
host:10.5.8.156

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request hash = 15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22
2020-03-30 14:45:04.323 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22
2020-03-30 14:45:04.323 7f4bdbcf0700 10 date_k    = b24064027f23cd98c3255b42b2ee511d907663f0e29c42e15e8f2b843a231cbc
2020-03-30 14:45:04.323 7f4bdbcf0700 10 region_k  = 4a1865641ecc0ab67a6dced8b6d09822752c873b8a5d19bacb6139f10a01e7dd
2020-03-30 14:45:04.323 7f4bdbcf0700 10 service_k = a2eff4dc70d9f170de38439e4732287358c3ebd821d9d354c188ca409a09fbbb
2020-03-30 14:45:04.323 7f4bdbcf0700 10 signing_k = 6be6df1acb55821c2571af97a21845caf277ce8007103d7bde1188c5b0403528
2020-03-30 14:45:04.323 7f4bdbcf0700 10 generated signature = 6ad1438a1eded893f0f89153efeaf00d97129d99bd4c2f9467096704f8cd7e57
2020-03-30 14:45:04.323 7f4bdbcf0700  5 Failed the auth strategy, reason=-2027
2020-03-30 14:45:04.323 7f4bdbcf0700 10 failed to authorize request
2020-03-30 14:45:04.327 7f4bdbcf0700  2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:op status=0
2020-03-30 14:45:04.327 7f4bdbcf0700  2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:http status=403

3. Upload with metadata successfully (add more header in curl command to work arround)

curl --request PUT --upload-file 1.jpg  -H 'Content-Type:image/jpeg' -H 'x-amz-acl:public-read' "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f" 

This is rgw log

2020-03-30 14:50:09.071 7f4bdbcf0700 10 credential scope = 20200330/us-east-1/s3/aws4_request
2020-03-30 14:50:09.071 7f4bdbcf0700 10 canonical headers format = content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

2020-03-30 14:50:09.071 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD
2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request = PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request hash = c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391
2020-03-30 14:50:09.075 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391

Rgw not add metadata from url in

http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09flink

to canonical request.
I think rgw need add parameter from query string in signed url to canonical request.

Thanks.

History

#1 Updated by Casey Bodley 3 months ago

  • Assignee set to Or Friedmann

#2 Updated by Or Friedmann 3 months ago

Hi Hoan,

I ran this script:

import boto3
import botocore

s3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

then run:

curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"

It works

Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)

#3 Updated by hoan nv 3 months ago

Or Friedmann wrote:

Hi Hoan,

I ran this script:

import boto3
import botocore

s3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

then run:

curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"

It works

Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)

I was tested this script with Amazon S3, it wwork. Curl command don't need add content-type and x-amz-acl headers to curl request. I think rgw should work same as Amazon S3.

#4 Updated by Or Friedmann 2 months ago

curl --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

curl -H "Content-Type: image/jpeg" -H "x-amz-acl: private" --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"

worked

It looks like it is mandatory in AWS too to send the signed headers.

#5 Updated by Or Friedmann 2 months ago

  • Status changed from New to Closed

Also available in: Atom PDF