Project

General

Profile

Bug #15369

New AWSv4 signature support doesn't work: causes s3cmd to get 403/SignatureDoesNotMatch

Added by Robin Johnson over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Urgent
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
rgw, aws4
Backport:
Regression:
Yes
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

s3cmd uses v4 signatures by default, and generates them differently than RGW does. I'd trust s3cmd at this point, because it works with AWS...

RGW is building the canonical URL incorrectly.

s3cmd:

# s3cmd -c s3cfg-rgw ls s3://pewpew --debug
DEBUG: Unicodising 's3://pewpew' using UTF-8
DEBUG: Command: ls
DEBUG: Bucket 's3://pewpew':
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v4
DEBUG: get_hostname(pewpew): pewpew.CENSORED
DEBUG: canonical_headers = host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160404T214839Z

DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160404T214839Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=JZXZ88D0YDP4ZACU0U78/20160404/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=2e3c40e01716aa7d618644db6d033a96fc354257d26b864c46e1ee88b3f530a9', 'x-amz-date': '20160404T214839Z'}

rgw logs:

2016-04-04 14:48:39.381801 7f0bcffff700  1 ====== starting new request req=0x7f0bcfff97d0 =====
2016-04-04 14:48:39.381824 7f0bcffff700  2 req 4812:0.000024::GET /::initializing for trans_id = tx0000000000000000012cc-005702e137-1859-default
2016-04-04 14:48:39.381830 7f0bcffff700 10 host=pewpew.CENSORED
2016-04-04 14:48:39.381838 7f0bcffff700 20 subdomain=pewpew domain=CENSORED in_hosted_domain=1 in_hosted_domain_s3website=0
2016-04-04 14:48:39.381855 7f0bcffff700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2016-04-04 14:48:39.381861 7f0bcffff700 10 meta>> HTTP_X_AMZ_DATE
2016-04-04 14:48:39.381867 7f0bcffff700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-04-04 14:48:39.381870 7f0bcffff700 10 x>> x-amz-date:20160404T214839Z
2016-04-04 14:48:39.381896 7f0bcffff700 20 get_handler handler=25RGWHandler_REST_Bucket_S3
2016-04-04 14:48:39.381902 7f0bcffff700 10 handler=25RGWHandler_REST_Bucket_S3
2016-04-04 14:48:39.381904 7f0bcffff700  2 req 4812:0.000104:s3:GET /::getting op 0
2016-04-04 14:48:39.381912 7f0bcffff700 10 op=25RGWListBucket_ObjStore_S3
2016-04-04 14:48:39.381913 7f0bcffff700  2 req 4812:0.000113:s3:GET /:list_bucket:authorizing
2016-04-04 14:48:39.381924 7f0bcffff700 10 v4 signedheaders format = host;x-amz-content-sha256;x-amz-date
2016-04-04 14:48:39.381928 7f0bcffff700 10 v4 signature format = 2e3c40e01716aa7d618644db6d033a96fc354257d26b864c46e1ee88b3f530a9
2016-04-04 14:48:39.381936 7f0bcffff700 10 v4 credential format = JZXZ88D0YDP4ZACU0U78/20160404/default/s3/aws4_request
2016-04-04 14:48:39.381939 7f0bcffff700 10 access key id = JZXZ88D0YDP4ZACU0U78
2016-04-04 14:48:39.381941 7f0bcffff700 10 credential scope = 20160404/default/s3/aws4_request
2016-04-04 14:48:39.381988 7f0bcffff700 10 canonical headers format = host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160404T214839Z

2016-04-04 14:48:39.382028 7f0bcffff700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-04-04 14:48:39.382051 7f0bcffff700 10 canonical request = GET
/pewpew/
delimiter=%2F
host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160404T214839Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-04-04 14:48:39.382053 7f0bcffff700 10 canonical request hash = 9914877ef73e5ce2c41e571c9711aae9f329329e74174aeb3e163847abd24fb1
2016-04-04 14:48:39.382057 7f0bcffff700 10 string to sign = AWS4-HMAC-SHA256
20160404T214839Z
20160404/default/s3/aws4_request
9914877ef73e5ce2c41e571c9711aae9f329329e74174aeb3e163847abd24fb1
2016-04-04 14:48:39.382112 7f0bcffff700 10 date_k        = fe70e1eb03793f7ee8083a6c3c077b139fadca8d2acda7a2e90c443372dc1ba1
2016-04-04 14:48:39.382146 7f0bcffff700 10 region_k      = 058a25d53713b648eb0596463b353d7a6a8b25946809572a2e25b117688b67df
2016-04-04 14:48:39.382177 7f0bcffff700 10 service_k     = 954c1dcb71d4e11240de5da701cab84a2dd9d4704507da8a938d4f1c42ba0478
2016-04-04 14:48:39.382214 7f0bcffff700 10 signing_k     = 18fbccfcff43a4650cab1b6b6ed5322f4809e9e03e3147e2189d247d1a2c5964
2016-04-04 14:48:39.382250 7f0bcffff700 10 signature_k   = f6c8e7fa61af982556d96dd898d8bf9cb8a3695478ac381a8c0057cdf73d25a0
2016-04-04 14:48:39.382255 7f0bcffff700 10 new signature = f6c8e7fa61af982556d96dd898d8bf9cb8a3695478ac381a8c0057cdf73d25a0
2016-04-04 14:48:39.382256 7f0bcffff700 10 ----------------------------- Verifying signatures
2016-04-04 14:48:39.382257 7f0bcffff700 10 Signature     = 2e3c40e01716aa7d618644db6d033a96fc354257d26b864c46e1ee88b3f530a9
2016-04-04 14:48:39.382259 7f0bcffff700 10 New Signature = f6c8e7fa61af982556d96dd898d8bf9cb8a3695478ac381a8c0057cdf73d25a0
2016-04-04 14:48:39.382260 7f0bcffff700 10 -----------------------------
2016-04-04 14:48:39.382265 7f0bcffff700 10 failed to authorize request
2016-04-04 14:48:39.382267 7f0bcffff700 20 handler->ERRORHANDLER: err_no=-2027 new_err_no=-2027


Related issues

Related to rgw - Bug #15358: rgw signature mismatch with escaped characters in url query portion Resolved 04/01/2016

History

#1 Updated by Javier M. Mellid over 6 years ago

  • Assignee set to Javier M. Mellid

Hi Robin, thanks for reporting. Could it be a dupe of #15358? If so, it could be a bug with escaped characters. I am watching the '%2F' character in both ticket reports. I will have a look.

#2 Updated by Javier M. Mellid over 6 years ago

Robin, I just asked for PR to fix http://tracker.ceph.com/issues/15358 It is under review now.

PR at: https://github.com/ceph/ceph/pull/8445

#3 Updated by Nathan Cutler over 6 years ago

  • Related to Bug #15358: rgw signature mismatch with escaped characters in url query portion added

#4 Updated by Robin Johnson over 6 years ago

@javier:
I don't think that's going to fix it, but I'll test quickly for you.

#5 Updated by Robin Johnson over 6 years ago

@javier:
Your patch does NOT fix this bug.

The two sides still disagree on the canonical URI.
s3cmd:

DEBUG: Canonical Request:
GET
/
delimiter=%2F
host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160405T203849Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------

rgw:

2016-04-05 13:38:49.938962 7f99777fe700 10 canonical request = GET
/pewpew/
delimiter=%2F
host:pewpew.CENSORED
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160405T203849Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-04-05 13:38:49.938966 7f99777fe700 10 canonical request hash = 4e6789cb0dbe6fc82743717f17570832083a056a0ed8e9fb329083c4ab542ca8

#6 Updated by Javier M. Mellid over 6 years ago

@robin thanks for confirming. That 'delimiter=%2F' line in the logs was a suspicious looking character :)

I am not very familiar with s3cmd but having a look in the logs it looks it uses part of the name of the host as a bucket. Are you using Subdomain calling instead of Ordinary calling? If so, would it be possible configuring s3cmd to use Ordinary calling?

In our testing suite we are using 'calling_format = boto.s3.connection.OrdinaryCallingFormat()' to get this calling format with boto2

BTW, it would be great if we can reproduce the bug in python in order to add s3-test coverage in the short if needed. Bear in mind we are growing the AWS4 implementation from boto2 and the s3-tests at the end.

#7 Updated by Javier M. Mellid over 6 years ago

I was able to reproduce the bug with 's3cmd 1.6.1' and subdomain calling. The bug lives in one of the RGW REST paths where the name of the bucket shipping as part of the host is included at the beginning of the request uri under Subdomain calling. The AWS4 algorithm should use the original request uri (the one used to hash in the client side) instead of the modified one.

I implemented a quick fix and it looks working ok with s3cmd (listing/creating/uploading/retrieving). s3-tests also pass on my side. I will ask for PR in a while.

#9 Updated by Javier M. Mellid over 6 years ago

@robin can you confirm this PR fixes the issue with s3cmd? The following tests under s3cmd 1.6.1 are working on my side:

s3cmd -c s3test.cfg mb s3://mybucket
s3cmd -c s3test.cfg put ~/test.txt s3://mybucket/myobject
s3cmd -c s3test.cfg ls s3://mybucket
s3cmd -c s3test.cfg get s3://mybucket/myobject

Request log...

PUT / HTTP/1.1
Host: mybucket.localhost:8000
Accept-Encoding: identity
Content-Length: 0
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=ABFIOVKTAZLOAF43WNQA/20160406/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=5df5b1cec828eabc95fd828f38a27f3b9dcc6ad7dde41eeea960b4920c42795f
x-amz-date: 20160406T134538Z

HTTP/1.1 200 OK
x-amz-request-id: tx000000000000000000001-0057051302-1016-default
Content-Length: 0
Date: Wed, 06 Apr 2016 13:45:47 GMT

Thanks!

#10 Updated by Javier M. Mellid over 6 years ago

  • Status changed from 12 to Fix Under Review

#11 Updated by Robin Johnson over 6 years ago

@jmunhoz:
Fix confirmed, thanks! Please merge.

#12 Updated by Sage Weil over 6 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF