Bug #44804: Signed Url generated with metadata alway return SignatureNotMatch - rgw - Ceph

Actions

Copy link

Bug #44804

closed

Signed Url generated with metadata alway return SignatureNotMatch

Added by hoan nv about 4 years ago. Updated almost 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Or Friedmann

Target version:

Ceph - v13.2.7

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v13.2.7, Ceph - v14.2.7

ceph-qa-suite:

rgw

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Hi all.

1. Upload without metadata (successfully)
I use my code to generate url

s3client = boto3.client(service_name='s3',
                  aws_access_key_id=access_key,
                  aws_secret_access_key=secret_key,
                  endpoint_url=endpoint_url, use_ssl=False, verify=False,
                  config=Config(signature_version='s3v4'))
bucket_name = 'bucket3'
s3client.create_bucket(Bucket=bucket_name)
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

I have presigned url to put object

http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b

Then i use curl

curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20200330T073700Z&X-Amz-Signature=b3727e6826c14fa4fca3ba10af3185eef8d891ecc865586831965a25b626997b"

I upload file successfully.

2. Upload with metadata (403 SignatureNotMatch)

But I add more header metadata

url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg','ContentType':'image/jpeg','ACL':'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

Request return 403 SignatureNotMatch

This is log debug boto3

2020-03-30 14:44:05,027 botocore.auth [DEBUG] Calculating signature using v4 auth.
2020-03-30 14:44:05,028 botocore.auth [DEBUG] CanonicalRequest:
PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:44:05,028 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391
2020-03-30 14:44:05,029 botocore.auth [DEBUG] Signature:
4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f
http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f

Then i use curl command to upload

curl --request PUT --upload-file 1.jpg "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f" 

It rerturn :
<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000038-005e81a380-2122-default</RequestId><HostId>2122-default-default</HostId>

This is rgw debug log.

2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical headers format = host:10.5.8.156

2020-03-30 14:45:04.323 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD
2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request = PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
host:10.5.8.156

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:45:04.323 7f4bdbcf0700 10 canonical request hash = 15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22
2020-03-30 14:45:04.323 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
15bf4c3b375c85b3a785a62ed1961f545f99b0a73571118953196b4272f12b22
2020-03-30 14:45:04.323 7f4bdbcf0700 10 date_k    = b24064027f23cd98c3255b42b2ee511d907663f0e29c42e15e8f2b843a231cbc
2020-03-30 14:45:04.323 7f4bdbcf0700 10 region_k  = 4a1865641ecc0ab67a6dced8b6d09822752c873b8a5d19bacb6139f10a01e7dd
2020-03-30 14:45:04.323 7f4bdbcf0700 10 service_k = a2eff4dc70d9f170de38439e4732287358c3ebd821d9d354c188ca409a09fbbb
2020-03-30 14:45:04.323 7f4bdbcf0700 10 signing_k = 6be6df1acb55821c2571af97a21845caf277ce8007103d7bde1188c5b0403528
2020-03-30 14:45:04.323 7f4bdbcf0700 10 generated signature = 6ad1438a1eded893f0f89153efeaf00d97129d99bd4c2f9467096704f8cd7e57
2020-03-30 14:45:04.323 7f4bdbcf0700  5 Failed the auth strategy, reason=-2027
2020-03-30 14:45:04.323 7f4bdbcf0700 10 failed to authorize request
2020-03-30 14:45:04.327 7f4bdbcf0700  2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:op status=0
2020-03-30 14:45:04.327 7f4bdbcf0700  2 req 56:0.00399995s:s3:PUT /bucket3/1.jpg:put_obj:http status=403

3. Upload with metadata successfully (add more header in curl command to work arround)

curl --request PUT --upload-file 1.jpg  -H 'Content-Type:image/jpeg' -H 'x-amz-acl:public-read' "http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09f"

This is rgw log

2020-03-30 14:50:09.071 7f4bdbcf0700 10 credential scope = 20200330/us-east-1/s3/aws4_request
2020-03-30 14:50:09.071 7f4bdbcf0700 10 canonical headers format = content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

2020-03-30 14:50:09.071 7f4bdbcf0700 10 payload request hash = UNSIGNED-PAYLOAD
2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request = PUT
/bucket3/1.jpg
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200330T074405Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl
content-type:image/jpeg
host:10.5.8.156
x-amz-acl:public-read

content-type;host;x-amz-acl
UNSIGNED-PAYLOAD
2020-03-30 14:50:09.075 7f4bdbcf0700 10 canonical request hash = c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391
2020-03-30 14:50:09.075 7f4bdbcf0700 10 string to sign = AWS4-HMAC-SHA256
20200330T074405Z
20200330/us-east-1/s3/aws4_request
c8ee46724d869e942869f20b320b83b4e2ea17dca25055d75805ff891ca72391

Rgw not add metadata from url in

http://10.5.8.156:80/bucket3/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=74AWMS8A1AP0MZYKQO0W%2F20200330%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Date=20200330T074405Z&X-Amz-Signature=4052d1f932ead5d54133a50880fb8e8b3420b3f9e013873bb4bf33c744a8b09flink

to canonical request.
I think rgw need add parameter from query string in signed url to canonical request.

Thanks.

Actions

Copy link

Updated by Casey Bodley about 4 years ago

Assignee set to Or Friedmann

Actions

Copy link

Updated by Or Friedmann about 4 years ago

Hi Hoan,

I ran this script:

import boto3
import botocore

s3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

then run:

curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"

It works

Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)

Actions

Copy link

Updated by hoan nv about 4 years ago

Or Friedmann wrote:

Hi Hoan,

I ran this script:

import boto3
import botocore

s3client = boto3.client(service_name='s3',
aws_access_key_id='0555b35654ad1656d804',
aws_secret_access_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==',
endpoint_url='http://127.0.0.1:8000', use_ssl=False, verify=False,
config=botocore.config.Config(signature_version='s3v4')
)
bucket_name = 'hello'
url = s3client.generate_presigned_url('put_object', Params={'Bucket':bucket_name, 'Key':'1.jpg', 'ContentType': 'image/jpeg' , 'ACL': 'public-read'}, ExpiresIn=3600, HttpMethod='PUT')
print(url)

then run:

curl -H "content-type: image/jpeg" -H "x-amz-acl: public-read" --request PUT --upload-file /etc/hosts "<the url from the python3 sccript>"

It works

Well RGW is not able to add the headers as the presigned url is being created by boto3 and not by the RGW (you can see in the logs that not request has sent to the RGW after running the python script)

I was tested this script with Amazon S3, it wwork. Curl command don't need add content-type and x-amz-acl headers to curl request. I think rgw should work same as Amazon S3.

Actions

Copy link

Updated by Or Friedmann almost 4 years ago

curl --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

curl -H "Content-Type: image/jpeg" -H "x-amz-acl: private" --request PUT --upload-file 1.jpg "https://s3.amazonaws.com/testsignedurls/1.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<access_key>%2F20200430%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200430T090739Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost%3Bx-amz-acl&X-Amz-Signature=<signature>"

worked

It looks like it is mandatory in AWS too to send the signed headers.

Actions

Copy link

Updated by Or Friedmann almost 4 years ago

Status changed from New to Closed

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » rgw

Custom queries

Bug #44804

Signed Url generated with metadata alway return SignatureNotMatch

Updated by Casey Bodley about 4 years ago

Updated by Or Friedmann about 4 years ago

Updated by hoan nv about 4 years ago

Updated by Or Friedmann almost 4 years ago

Updated by Or Friedmann almost 4 years ago