Bug #23470
closedpresigned URL for PUT with metadata fails: SignatureDoesNotMatch
0%
Description
- Pre-signed URLs for PUT without metadata works fine.
- Pre-signed URL for PUT with metadata gives SignatureDoesNotMatch
Happens for both v2 & v4 signature. Busy confirming against AWS.
# generate presigned URLs for PUT upload with & without metadata. host = 'objects-us-west-1.dream.io' aws_access_key_id = ... aws_secret_access_key = ... bucket = ... key = ... expires = 900 import boto from boto.s3.connection import S3Connection conn = S3Connection(host=host,aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key, is_secure=False, debug=3, calling_format = 'boto.s3.connection.OrdinaryCallingFormat') bucket_obj = boto.s3.bucket.Bucket(connection=conn, name=bucket) key_obj = boto.s3.key.Key(bucket_obj, key) print(key_obj.generate_url(expires, method='PUT', force_http=True)) key_obj.metadata['md'] = 'md' print(key_obj.generate_url(expires, method='PUT', force_http=True)) # do upload with: curl -T testfile -v $URL
Updated by Robin Johnson about 6 years ago
- Subject changed from presigned URL for PUT with metadata fails to presigned URL for PUT with metadata fails: SignatureDoesNotMatch
Updated by Robin Johnson about 6 years ago
# AWSv2 upload result, no metadata vs metadata ==== DOING http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=B%2BKgCpV%2F%2Fas5vpR1QT8hxsy%2BOhU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY + curl -v -T ./run.sh 'http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=B%2BKgCpV%2F%2Fas5vpR1QT8hxsy%2BOhU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY' --header Expect: * Trying 2607:f298:4:143:acce:55:2:1... * TCP_NODELAY set * Connected to objects-us-west-1.dream.io (2607:f298:4:143:acce:55:2:1) port 80 (#0) > PUT /site-monitor-congress/testing-python?Signature=B%2BKgCpV%2F%2Fas5vpR1QT8hxsy%2BOhU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY HTTP/1.1 > Host: objects-us-west-1.dream.io > User-Agent: curl/7.55.0 > Accept: */* > Content-Length: 754 > * We are completely uploaded and fine < HTTP/1.1 200 OK < Content-Length: 0 < ETag: "15cbe5d37f9c196b01314d9e81b15cbe" < Accept-Ranges: bytes < x-amz-request-id: tx000000000000000474d43-005abb213d-20e4b130-default < Date: Wed, 28 Mar 2018 04:59:42 GMT < * Connection #0 to host objects-us-west-1.dream.io left intact ==== DOING http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=pDeVazvoJELg3gU4vQ0ZTr%2BKtoU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY&x-amz-meta-md=md + curl -v -T ./run.sh 'http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=pDeVazvoJELg3gU4vQ0ZTr%2BKtoU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY&x-amz-meta-md=md' --header Expect: * Trying 2607:f298:4:143:acce:55:2:1... * TCP_NODELAY set * Connected to objects-us-west-1.dream.io (2607:f298:4:143:acce:55:2:1) port 80 (#0) > PUT /site-monitor-congress/testing-python?Signature=pDeVazvoJELg3gU4vQ0ZTr%2BKtoU%3D&Expires=1522214081&AWSAccessKeyId=MY-ACCESS-KEY&x-amz-meta-md=md HTTP/1.1 > Host: objects-us-west-1.dream.io > User-Agent: curl/7.55.0 > Accept: */* > Content-Length: 754 > * We are completely uploaded and fine < HTTP/1.1 403 Forbidden < Content-Length: 202 < x-amz-request-id: tx000000000000000474d68-005abb213e-20e4b103-default < Accept-Ranges: bytes < Content-Type: application/xml < Date: Wed, 28 Mar 2018 04:59:42 GMT < * Connection #0 to host objects-us-west-1.dream.io left intact <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000474d68-005abb213e-20e4b103-default</RequestId><HostId>20e4b103-default-default</HostId></Error> # AWSv4 upload result, no metadata vs metadata ==== DOING http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=caa3f2955f3e8e818274d482c07c8fe78571a23f08bd7f5fd677aa165559acb8&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request + curl -v -T ./run.sh 'http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=caa3f2955f3e8e818274d482c07c8fe78571a23f08bd7f5fd677aa165559acb8&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request' --header Expect: * Trying 2607:f298:4:143:acce:55:2:1... * TCP_NODELAY set * Connected to objects-us-west-1.dream.io (2607:f298:4:143:acce:55:2:1) port 80 (#0) > PUT /site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=caa3f2955f3e8e818274d482c07c8fe78571a23f08bd7f5fd677aa165559acb8&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request HTTP/1.1 > Host: objects-us-west-1.dream.io > User-Agent: curl/7.55.0 > Accept: */* > Content-Length: 722 > * We are completely uploaded and fine < HTTP/1.1 200 OK < Content-Length: 0 < ETag: "ed905b1347724baecc63e319ee477e8c" < Accept-Ranges: bytes < x-amz-request-id: tx0000000000000006c1aab-005abb1dc5-20d426cb-default < Date: Wed, 28 Mar 2018 04:44:54 GMT < * Connection #0 to host objects-us-west-1.dream.io left intact ==== DOING http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-meta-md&X-Amz-Signature=687f790bbf6970fa24d5261c1846a8bd1c35d74c9d1aaa6bc63409b8c4b785b1&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request + curl -v -T ./run.sh 'http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-meta-md&X-Amz-Signature=687f790bbf6970fa24d5261c1846a8bd1c35d74c9d1aaa6bc63409b8c4b785b1&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request' --header Expect: * Trying 2607:f298:4:143:acce:55:2:1... * TCP_NODELAY set * Connected to objects-us-west-1.dream.io (2607:f298:4:143:acce:55:2:1) port 80 (#0) > PUT /site-monitor-congress/testing-python?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-meta-md&X-Amz-Signature=687f790bbf6970fa24d5261c1846a8bd1c35d74c9d1aaa6bc63409b8c4b785b1&X-Amz-Date=20180328T044452Z&X-Amz-Credential=MY-ACCESS-KEY%2F20180328%2Fobjects-us-west-1%2Fs3%2Faws4_request HTTP/1.1 > Host: objects-us-west-1.dream.io > User-Agent: curl/7.55.0 > Accept: */* > Content-Length: 722 > * We are completely uploaded and fine < HTTP/1.1 403 Forbidden < Content-Length: 202 < x-amz-request-id: tx00000000000000046fa1a-005abb1dc6-20e4b103-default < Accept-Ranges: bytes < Content-Type: application/xml < Date: Wed, 28 Mar 2018 04:44:54 GMT < * Connection #0 to host objects-us-west-1.dream.io left intact <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx00000000000000046fa1a-005abb1dc6-20e4b103-default</RequestId><HostId>20e4b103-default-default</HostId></Error>
Updated by Prashant D about 6 years ago
Could not get much time today but got hold of rgw debug logs for the issue.
2018-03-28 07:12:01.410905 7f99ab8af700 1 ====== starting new request req=0x7f99ab8a9190 =====
2018-03-28 07:12:01.410951 7f99ab8af700 2 req 1:0.000047::PUT /my-new-bucket/tesrt::initializing for trans_id = tx000000000000000000001-005abb7881-1396-us-east
2018-03-28 07:12:01.410973 7f99ab8af700 10 rgw api priority: s3=5 s3website=4
2018-03-28 07:12:01.410976 7f99ab8af700 10 host=10.xx.xx.xx
2018-03-28 07:12:01.410981 7f99ab8af700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2018-03-28 07:12:01.410987 7f99ab8af700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/my-new-bucket/tesrt
2018-03-28 07:12:01.411118 7f99ab8af700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2018-03-28 07:12:01.411133 7f99ab8af700 10 handler=22RGWHandler_REST_Obj_S3
2018-03-28 07:12:01.411136 7f99ab8af700 2 req 1:0.000232:s3:PUT /my-new-bucket/tesrt::getting op 1
2018-03-28 07:12:01.411202 7f99ab8af700 10 op=21RGWPutObj_ObjStore_S3
2018-03-28 07:12:01.411205 7f99ab8af700 2 req 1:0.000302:s3:PUT /my-new-bucket/tesrt:put_obj:verifying requester
2018-03-28 07:12:01.411216 7f99ab8af700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2018-03-28 07:12:01.411220 7f99ab8af700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2018-03-28 07:12:01.411233 7f99ab8af700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2018-03-28 07:12:01.411236 7f99ab8af700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2018-03-28 07:12:01.411276 7f99ab8af700 10 v4 credential format = IRQGSAMBKKO4YUN7T375/20180328/us-east/s3/aws4_request
2018-03-28 07:12:01.411282 7f99ab8af700 10 access key id = IRQGSAMBKKO4YUN7T375
2018-03-28 07:12:01.411284 7f99ab8af700 10 credential scope = 20180328/us-east/s3/aws4_request
2018-03-28 07:12:01.411302 7f99ab8af700 10 warning env var not available
2018-03-28 07:12:01.411308 7f99ab8af700 10 canonical headers format = host:10.xx.xx.xx:8080
2018-03-28 07:12:01.411352 7f99ab8af700 10 payload request hash = UNSIGNED-PAYLOAD
2018-03-28 07:12:01.411389 7f99ab8af700 10 canonical request = PUT
/my-new-bucket/tesrt
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=IRQGSAMBKKO4YUN7T375%2F20180328%2Fus-east%2Fs3%2Faws4_request&X-Amz-Date=20180328T111129Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-meta-md
host:10.xx.xx.xx:8080
host;x-amz-meta-md
UNSIGNED-PAYLOAD
2018-03-28 07:12:01.411394 7f99ab8af700 10 canonical request hash = 7b179623a986ebd0cc4dad47d6468e20a5cec617bd3ab4ae4ec68ea8218980db
2018-03-28 07:12:01.411417 7f99ab8af700 10 string to sign = AWS4-HMAC-SHA256
20180328T111129Z
20180328/us-east/s3/aws4_request
7b179623a986ebd0cc4dad47d6468e20a5cec617bd3ab4ae4ec68ea8218980db
2018-03-28 07:12:01.411459 7f99ab8af700 20 get_system_obj_state: rctx=0x7f99ab8a7730 obj=us-east.rgw.meta:users.keys:IRQGSAMBKKO4YUN7T375 state=0x560097d16ee0 s->prefetch_data=0
2018-03-28 07:12:01.411518 7f99ab8af700 10 cache get: name=us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 : miss
2018-03-28 07:12:01.413345 7f99ab8af700 10 cache put: name=us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 info.flags=0x6
2018-03-28 07:12:01.413370 7f99ab8af700 10 adding us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 to cache LRU end
2018-03-28 07:12:01.413378 7f99ab8af700 20 get_system_obj_state: s->obj_tag was set empty
2018-03-28 07:12:01.413384 7f99ab8af700 10 cache get: name=us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 : type miss (requested=0x1, cached=0x6)
2018-03-28 07:12:01.413388 7f99ab8af700 20 rados->read ofs=0 len=524288
2018-03-28 07:12:01.414462 7f99ab8af700 20 rados->read r=0 bl.length=9
2018-03-28 07:12:01.414501 7f99ab8af700 10 cache put: name=us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 info.flags=0x1
2018-03-28 07:12:01.414508 7f99ab8af700 10 moving us-east.rgw.meta+users.keys+IRQGSAMBKKO4YUN7T375 to cache LRU end
2018-03-28 07:12:01.414524 7f99ab8af700 20 get_system_obj_state: rctx=0x7f99ab8a7370 obj=us-east.rgw.meta:users.uid:rhsys state=0x560097d173c0 s->prefetch_data=0
2018-03-28 07:12:01.414533 7f99ab8af700 10 cache get: name=us-east.rgw.meta+users.uid+rhsys : miss
2018-03-28 07:12:01.419140 7f99ab8af700 10 cache put: name=us-east.rgw.meta+users.uid+rhsys info.flags=0x16
2018-03-28 07:12:01.419192 7f99ab8af700 10 adding us-east.rgw.meta+users.uid+rhsys to cache LRU end
2018-03-28 07:12:01.419203 7f99ab8af700 20 get_system_obj_state: s->obj_tag was set empty
2018-03-28 07:12:01.419211 7f99ab8af700 10 cache get: name=us-east.rgw.meta+users.uid+rhsys : type miss (requested=0x11, cached=0x16)
2018-03-28 07:12:01.419234 7f99ab8af700 20 rados->read ofs=0 len=524288
2018-03-28 07:12:01.423446 7f99ab8af700 20 rados->read r=0 bl.length=486
2018-03-28 07:12:01.423490 7f99ab8af700 10 cache put: name=us-east.rgw.meta+users.uid+rhsys info.flags=0x11
2018-03-28 07:12:01.423499 7f99ab8af700 10 moving us-east.rgw.meta+users.uid+rhsys to cache LRU end
2018-03-28 07:12:01.423542 7f99ab8af700 10 chain_cache_entry: cache_locator=us-east.rgw.meta+users.uid+rhsys
2018-03-28 07:12:01.423733 7f99ab8af700 10 date_k = bb2394b68d27fd96fc3de3537edca9dc5f84cdb1219be6ba6c649f888e008823
2018-03-28 07:12:01.423759 7f99ab8af700 10 region_k = cf926bdd06b299330435bfe18c60019044c5637b7807e80cfafa8d2cab98cd01
2018-03-28 07:12:01.423769 7f99ab8af700 10 service_k = 31ba8dfbf7fea4e784219bede9cb799127962d9add61e4fd0a6c358bfd98c5af
2018-03-28 07:12:01.423784 7f99ab8af700 10 signing_k = 5b138b6949731c37b53b7aaee355804283a32f2d532a136e720cd9671c90a2f2
2018-03-28 07:12:01.423824 7f99ab8af700 10 generated signature = f994f18ddd15bb4eb27b52b66c2aca6856ab75314a5623acde86a022be46df0f
2018-03-28 07:12:01.423834 7f99ab8af700 15 string_to_sign=AWS4-HMAC-SHA256
20180328T111129Z
20180328/us-east/s3/aws4_request
7b179623a986ebd0cc4dad47d6468e20a5cec617bd3ab4ae4ec68ea8218980db
2018-03-28 07:12:01.423853 7f99ab8af700 15 server signature=f994f18ddd15bb4eb27b52b66c2aca6856ab75314a5623acde86a022be46df0f
2018-03-28 07:12:01.423855 7f99ab8af700 15 client signature=4623b24402b605ac6244b43cfc20ce24754853b88eb83383a226a7a4713c6d58
2018-03-28 07:12:01.423859 7f99ab8af700 15 compare=-50
2018-03-28 07:12:01.423865 7f99ab8af700 20 rgw::auth::s3::LocalEngine denied with reason=-2027
2018-03-28 07:12:01.423868 7f99ab8af700 20 rgw::auth::s3::AWSAuthStrategy denied with reason=-2027
2018-03-28 07:12:01.423872 7f99ab8af700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2018-03-28 07:12:01.423876 7f99ab8af700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2018-03-28 07:12:01.423884 7f99ab8af700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2018-03-28 07:12:01.423885 7f99ab8af700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2018-03-28 07:12:01.423899 7f99ab8af700 10 v4 credential format = IRQGSAMBKKO4YUN7T375/20180328/us-east/s3/aws4_request
2018-03-28 07:12:01.423902 7f99ab8af700 10 access key id = IRQGSAMBKKO4YUN7T375
2018-03-28 07:12:01.423904 7f99ab8af700 10 credential scope = 20180328/us-east/s3/aws4_request
2018-03-28 07:12:01.423919 7f99ab8af700 10 warning env var not available
2018-03-28 07:12:01.423924 7f99ab8af700 10 canonical headers format = host:10.xx.xx.xx:8080:8080
2018-03-28 07:12:01.423953 7f99ab8af700 10 payload request hash = UNSIGNED-PAYLOAD
2018-03-28 07:12:01.423974 7f99ab8af700 10 canonical request = PUT
/my-new-bucket/tesrt
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=IRQGSAMBKKO4YUN7T375%2F20180328%2Fus-east%2Fs3%2Faws4_request&X-Amz-Date=20180328T111129Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host%3Bx-amz-meta-md
host:10.xx.xx.xx:8080:8080
host;x-amz-meta-md
UNSIGNED-PAYLOAD
2018-03-28 07:12:01.423977 7f99ab8af700 10 canonical request hash = c32fd54e28f603d19784259ff747fa88132dbe24d179321f41746b4c2cc5b543
2018-03-28 07:12:01.424002 7f99ab8af700 10 string to sign = AWS4-HMAC-SHA256
20180328T111129Z
20180328/us-east/s3/aws4_request
Primarily looks like the rgw behavior according to spec and needs to verify from AWS point of view as well.
Updated by Robin Johnson about 6 years ago
@Prashant:
It does work against AWS us-east-1, I didn't test other regions.
I got boto to dump the StringToSign block for it, and it shows that the query parameter is being converted to a header for the signing.
DEBUG:boto:StringToSign:PUT 1522298886 /site-monitor-congress/testing-python http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=2pPQZ%2F%2Bdjkw6jwQlUFzEG5LsWEA%3D&Expires=1522298886&AWSAccessKeyId=REDACTED DEBUG:boto:StringToSign:PUT 1522298886 x-amz-meta-md:md /site-monitor-congress/testing-python http://objects-us-west-1.dream.io/site-monitor-congress/testing-python?Signature=6irCrWsoK9OvLo6fNfKzcHIAcFE%3D&Expires=1522298886&AWSAccessKeyId=REDACTED&x-amz-meta-md=md
Updated by Matt Benjamin about 6 years ago
- Status changed from New to In Progress
- Assignee set to Marcus Watts
- Priority changed from Normal to High
Updated by Matt Benjamin about 6 years ago
@Marcus Sorensen, could you take a look at this?
Updated by Matt Benjamin about 6 years ago
(marcus reported some progress debugging, will update)
Updated by Marcus Watts almost 6 years ago
Just a small update. After several dealing with several other mostly unrelated problems in master, I managed to reproduce this. It definitely doesn't incorporate the metadata strings into the signature like it should. In theory the fix should be fairly small.
Updated by Orit Wasserman almost 6 years ago
- Status changed from In Progress to 12
Updated by hoan nv over 5 years ago
I have some issue. My cluster running on mimic version.
Need backport to mimic version.
Thanks
Updated by Nathan Cutler over 5 years ago
- Backport changed from luminous to mimic
Updated by Casey Bodley over 5 years ago
Hi Marcus, can you share some details about the fix?
Updated by Matt Benjamin over 5 years ago
- Status changed from 12 to Fix Under Review
Updated by Matt Benjamin over 5 years ago
- Backport changed from mimic to mimic, luminous
Updated by Matt Benjamin over 5 years ago
- Assignee changed from Marcus Watts to Matt Benjamin
Updated by Casey Bodley over 5 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler over 5 years ago
- Copied to Backport #37558: mimic: presigned URL for PUT with metadata fails: SignatureDoesNotMatch added
Updated by Nathan Cutler over 5 years ago
- Copied to Backport #37559: luminous: presigned URL for PUT with metadata fails: SignatureDoesNotMatch added
Updated by Nathan Cutler about 5 years ago
- Status changed from Pending Backport to Resolved