Project

General

Profile

Actions
Copy link

Bug #16463

closed

AWS4 Presigned URL not accepted by RGW

Added by Frank Enderle almost 8 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Javier M. Mellid
Target version:
-
% Done:

0%

Spent time:
2:00 h
Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v10.2.2
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have a 3 node cluster (3 mons, 3 rgw, 36 osd) behind a haproxy instance.

I try to use mc (https://github.com/minio/mc) to generate a presigned GET request for an object. However the object is not retrieved from RGW instead it results in a 403 Forbidden.

A signed v2 URL works.

The mc command to reproduce the request are:

mc config host add rgw https://&lt;endpoint> <access_key> <secret_key> S3v4
mc share download rgw/<bucket>/<key>

It will output the signed URL to the console:

URL: https://***REDACTED***/test3/201623662.pdf
Expire: 7 days 0 hours 0 minutes 0 seconds
Share: https://***REDACTED***/test3/201623662.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***REDACTED***%2F20160623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160623T210052Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c6a1722018017f3c1a9c2fbedb0edd7a49b1fbc0d6e6db7fc02dd31f150d1469

If you use wget to receive the URL the following output occurs:

...
HTTP request sent, awaiting response... 403 Forbidden
2016-06-23 23:02:08 ERROR 403: Forbidden.

The debug rgw 20 log shows the following entries:

2016-06-23 21:02:58.903975 7f9626ffd700 20 RGWEnv::set(): HTTP_USER_AGENT: Wget/1.18 (darwin15.5.0)
2016-06-23 21:02:58.903992 7f9626ffd700 20 RGWEnv::set(): HTTP_ACCEPT: /*
2016-06-23 21:02:58.903996 7f9626ffd700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2016-06-23 21:02:58.904001 7f9626ffd700 20 RGWEnv::set(): HTTP_HOST: *REDACTED
2016-06-23 21:02:58.904003 7f9626ffd700 20 RGWEnv::set(): HTTP_X_FORWARDED_FOR: REDACTED
2016-06-23 21:02:58.904005 7f9626ffd700 20 RGWEnv::set(): HTTP_CONNECTION: close
2016-06-23 21:02:58.904008 7f9626ffd700 20 RGWEnv::set(): REQUEST_METHOD: GET
2016-06-23 21:02:58.904010 7f9626ffd700 20 RGWEnv::set(): REQUEST_URI: /test3/201623662.pdf
2016-06-23 21:02:58.904017 7f9626ffd700 20 RGWEnv::set(): QUERY_STRING: X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***REDACTED***%2F20160623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160623T210052Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c6a1722018017f3c1a9c2fbedb0edd7a49b1fbc0d6e6db7fc02dd31f150d1469
2016-06-23 21:02:58.904024 7f9626ffd700 20 RGWEnv::set(): REMOTE_USER:
2016-06-23 21:02:58.904026 7f9626ffd700 20 RGWEnv::set(): SCRIPT_URI: /test3/201623662.pdf
2016-06-23 21:02:58.904029 7f9626ffd700 20 RGWEnv::set(): SERVER_PORT: 7480
2016-06-23 21:02:58.904030 7f9626ffd700 20 HTTP_ACCEPT=*/*
2016-06-23 21:02:58.904032 7f9626ffd700 20 HTTP_ACCEPT_ENCODING=identity
2016-06-23 21:02:58.904033 7f9626ffd700 20 HTTP_CONNECTION=close
2016-06-23 21:02:58.904034 7f9626ffd700 20 HTTP_HOST=***REDACTED***
2016-06-23 21:02:58.904036 7f9626ffd700 20 HTTP_USER_AGENT=Wget/1.18 (darwin15.5.0)
2016-06-23 21:02:58.904037 7f9626ffd700 20 HTTP_X_FORWARDED_FOR=***REDACTED***
2016-06-23 21:02:58.904038 7f9626ffd700 20 QUERY_STRING=X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***REDACTED***%2F20160623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160623T210052Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=c6a1722018017f3c1a9c2fbedb0edd7a49b1fbc0d6e6db7fc02dd31f150d1469
2016-06-23 21:02:58.904045 7f9626ffd700 20 REMOTE_USER=
2016-06-23 21:02:58.904046 7f9626ffd700 20 REQUEST_METHOD=GET
2016-06-23 21:02:58.904047 7f9626ffd700 20 REQUEST_URI=/test3/201623662.pdf
2016-06-23 21:02:58.904048 7f9626ffd700 20 SCRIPT_URI=/test3/201623662.pdf
2016-06-23 21:02:58.904049 7f9626ffd700 20 SERVER_PORT=7480
2016-06-23 21:02:58.904052 7f9626ffd700 1 ====== starting new request req=0x7f9626ff7710 =====
2016-06-23 21:02:58.904076 7f9626ffd700 2 req 49:0.000024::GET /test3/201623662.pdf::initializing for trans_id = tx000000000000000000031-00576c4e82-58e5a-default
2016-06-23 21:02:58.904082 7f9626ffd700 10 host=***REDACTED***
2016-06-23 21:02:58.904089 7f9626ffd700 20 subdomain= domain=***REDACTED*** in_hosted_domain=1 in_hosted_domain_s3website=0
2016-06-23 21:02:58.904148 7f9626ffd700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2016-06-23 21:02:58.904154 7f9626ffd700 10 handler=22RGWHandler_REST_Obj_S3
2016-06-23 21:02:58.904156 7f9626ffd700 2 req 49:0.000104:s3:GET /test3/201623662.pdf::getting op 0
2016-06-23 21:02:58.904162 7f9626ffd700 10 op=21RGWGetObj_ObjStore_S3
2016-06-23 21:02:58.904164 7f9626ffd700 2 req 49:0.000112:s3:GET /test3/201623662.pdf:get_obj:authorizing
2016-06-23 21:02:58.904205 7f9626ffd700 10 v4 credential format = REDACTED/20160623/us-east-1/s3/aws4_request
2016-06-23 21:02:58.904209 7f9626ffd700 10 access key id = REDACTED
2016-06-23 21:02:58.904211 7f9626ffd700 10 credential scope = 20160623/us-east-1/s3/aws4_request
2016-06-23 21:02:58.904281 7f9626ffd700 10 canonical headers format = host:***REDACTED***:7480

2016-06-23 21:02:58.904285 7f9626ffd700 10 payload request hash = UNSIGNED-PAYLOAD
2016-06-23 21:02:58.904342 7f9626ffd700 10 canonical request = GET
/test3/201623662.pdf
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***REDACTED***%2F20160623%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160623T210052Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host
host:***REDACTED***:7480

host
UNSIGNED-PAYLOAD
2016-06-23 21:02:58.904344 7f9626ffd700 10 canonical request hash = 83a855651c9a6ff01f51bea38951fa43c7491b816f24d0f11cf8e92478c2ee2d
2016-06-23 21:02:58.904349 7f9626ffd700 10 string to sign = AWS4-HMAC-SHA256
20160623T210052Z
20160623/us-east-1/s3/aws4_request
83a855651c9a6ff01f51bea38951fa43c7491b816f24d0f11cf8e92478c2ee2d
2016-06-23 21:02:58.904412 7f9626ffd700 10 date_k = f5abcff2800eaff43c902bea85570d14d9eebf0dbcd956fd5209abcd6eaec1fe
2016-06-23 21:02:58.904451 7f9626ffd700 10 region_k = 61a870b01101b1bb02b82dac0b2b9b04c695ecbd26c5dbe733845e0a89676106
2016-06-23 21:02:58.904485 7f9626ffd700 10 service_k = 2c8cdecb7b9e65c6f2b549dbb1c04d42a2a01dc801656533fc79c12f525bcc4a
2016-06-23 21:02:58.904518 7f9626ffd700 10 signing_k = 82dfdfdec4dcdaaff20417ca180323851422f7afabca3d3a31a3d0f773209b1a
2016-06-23 21:02:58.904554 7f9626ffd700 10 signature_k = 00a40c296701da605f09b10aab4ae7c3f1f9aec4dc7b0a48fd22c7285dd38b29
2016-06-23 21:02:58.904559 7f9626ffd700 10 new signature = 00a40c296701da605f09b10aab4ae7c3f1f9aec4dc7b0a48fd22c7285dd38b29
2016-06-23 21:02:58.904560 7f9626ffd700 10 ----------------------------- Verifying signatures
2016-06-23 21:02:58.904561 7f9626ffd700 10 Signature = c6a1722018017f3c1a9c2fbedb0edd7a49b1fbc0d6e6db7fc02dd31f150d1469
2016-06-23 21:02:58.904565 7f9626ffd700 10 New Signature = 00a40c296701da605f09b10aab4ae7c3f1f9aec4dc7b0a48fd22c7285dd38b29
2016-06-23 21:02:58.904566 7f9626ffd700 10 -----------------------------
2016-06-23 21:02:58.904571 7f9626ffd700 10 failed to authorize request
2016-06-23 21:02:58.904573 7f9626ffd700 20 handler->ERRORHANDLER: err_no=-2027 new_err_no=-2027
2016-06-23 21:02:58.904705 7f9626ffd700 2 req 49:0.000653:s3:GET /test3/201623662.pdf:get_obj:op status=0
2016-06-23 21:02:58.904714 7f9626ffd700 2 req 49:0.000662:s3:GET /test3/201623662.pdf:get_obj:http status=403
2016-06-23 21:02:58.904719 7f9626ffd700 1 ====== req done req=0x7f9626ff7710 op status=0 http_status=403 ======
2016-06-23 21:02:58.904733 7f9626ffd700 20 process_request() returned -2027
2016-06-23 21:02:58.904766 7f9626ffd700 1 civetweb: 0x7f968c000bb0: REDACTED - - [23/Jun/2016:21:02:58 +0000] "GET /test3/201623662.pdf HTTP/1.1" 403 0 - Wget/1.18 (darwin15.5.0)
2016-06-23 21:02:59.586329 7f97b37fe700 2 RGWDataChangesLog::ChangesRenewThread: start

Related issues 1 (0 open — 1 closed)

Copied to rgw - Backport #20992: jewel: AWS4 Presigned URL not accepted by RGWResolvedRobin JohnsonActions
Actions
Copy link
 #1

Updated by Casey Bodley almost 8 years ago

  • Assignee set to Pritha Srivastava
Actions
Copy link
 #2

Updated by Javier M. Mellid almost 8 years ago

Thanks Fran for the detailed reporting. I was working this morning in the bug before being assigned and I was able to reproduce/identify the bug.

The bug is a well-known issue to work around the double port issue with boto2 (boto2 is being used as our main s3 client/library and it is used in s3-tests) You can read more on this bug here [1]

The problem is in the canonical request computation. Boto2 crafts the host using the port number twice while mc uses it once only.

The part of the code handling this stuff is at [2]:

if (using_qs && (token == "host")) {
      if (!port.empty() && port != "80") {
        token_value = token_value + ":" + port;
      } else if (!secure_port.empty() && secure_port != "443") {
        token_value = token_value + ":" + secure_port;
      }
}

To watch 'mc' working with qs requests we shouldn't concat the ":" + port, but it is like boto2 is working. If we modify this code we will break the boto2 signature and all cloud/devops libraries using boto2 currently (libcloud, ansible, etc)

Yehuda merged the patch handling ports with commit 033888bbd0e4d8d81358bf61a099276dddb5692b. Maybe it could make sense asking him to review.

By the way, Fran, I wonder if 'mc' works if you uses one default port (80 or 443) to bind the rgw server? With this possible workaround you should be able to skip the port concatenation and the canonical request hash should match. Maybe it is useful while the fix gets in place.

Casey, Pritha, feel free to assign me the bug if it makes sense.

--

[1] https://github.com/boto/boto/pull/3181
[2] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_s3.cc#L3667

Actions
Copy link
 #3

Updated by Nhật Khang Nguyễn almost 8 years ago 

I also receive a similar error when used pre-signed url with boto3. 
I also had read about https://github.com/ceph/ceph/commit/033888bbd0e4d8d81358bf61a099276dddb5692b. 
But I notice that with other port 443, it will run well. Here is what I get when running radosgw 10.2.2 with port 443s:

*ceph.config*:
rgw_frontends = "civetweb port=443s ssl_certificate=/etc/ceph/server.pem" 

*ceph-client.rgw.radosgw.log*:
//RGWEnv::set(): SERVER_PORT: 0
//RGWEnv::set(): SERVER_PORT_SECURE: 443
//HTTP_HOST=rgw.fshare.vn
//canonical headers format = host:xxx.yyy:0

*commit 033888bbd0e4d8d81358bf61a099276dddb5692b*:
+  string port = s->info.env->get("SERVER_PORT", "");
+  string secure_port = s->info.env->get("SERVER_PORT_SECURE", "");
//
+    if (using_qs && (token == "host")) {
+      if (!port.empty() && port != "80") {
+        token_value = token_value + ":" + port;
+      } else if (!secure_port.empty() && secure_port != "443") {
+        token_value = token_value + ":" + secure_port;
+      }
+    }
//

This means that when the secure_port=443 then port=0, the token_value = host value + ':' port and the result will be the host:xxx.yyy:0. 
It will make an error "failed to authorize the request".
I think it necessary to modify about env.set("SERVER_PORT"..) or env.set("SERVER_PORT_SECURE"..) in src/rgw/rgw_civetweb.cc=>RGWMongoose::init_env()
Actions
Copy link
 #4

Updated by Frank Enderle almost 8 years ago

Javier M. Mellid wrote:

By the way, Fran, I wonder if 'mc' works if you uses one default port (80 or 443) to bind the rgw server? With this possible workaround you should be able to skip the port concatenation and the canonical request hash should match. Maybe it is useful while the fix gets in place.

I just tried that and it didn't work. This is because I use the jewel release rpms and according to github in that commit the port is hardcoded to always be appended to the host token [1]:

    string token_value = string(t);
    if (using_qs && (token == "host"))
      token_value = token_value + ":" + port;
    canonical_hdrs_map[token] = rgw_trim_whitespace(token_value);

The commit to not append 80 and 443 was added in commit 033888b [2]

--
[1] https://github.com/ceph/ceph/blob/jewel/src/rgw/rgw_rest_s3.cc#L3503
[2] https://github.com/ceph/ceph/commit/033888bbd0e4d8d81358bf61a099276dddb5692b

Actions
Copy link
 #5

Updated by Javier M. Mellid almost 8 years ago

Pritha, I just pushed the patch I mentioned you yesterday. It is available at:

https://github.com/ceph/ceph/pull/10160

It is passing the test suite. I ran some manual tests with boto2 and mc clients.

The conf option works as expected with static (default values) and dynamic (via ceph.conf) configurations.

Actions
Copy link
 #6

Updated by Pritha Srivastava almost 8 years ago

  • Status changed from New to Fix Under Review
  • Assignee changed from Pritha Srivastava to Javier M. Mellid
Actions
Copy link
 #7

Updated by Charles Alva over 7 years ago

I got same results with Nginx as load balancer and TLS termination for Ceph RGWs v10.2.3. Looking forward for the bugfix. :)

...
20161117/us-east-1/s3/aws4_request
...
2016-11-16 17:43:15.675819 7ffa95fdb700 10 failed to authorize request
2016-11-16 17:43:15.675822 7ffa95fdb700 20 handler->ERRORHANDLER: err_no=-2027 new_err_no=-2027
2016-11-16 17:43:15.676030 7ffa95fdb700  2 req 4:0.001099:s3:PUT /test/:create_bucket:op status=0
2016-11-16 17:43:15.676037 7ffa95fdb700  2 req 4:0.001107:s3:PUT /test/:create_bucket:http status=403
2016-11-16 17:43:15.676050 7ffa95fdb700  1 ====== req done req=0x7ffa95fd57d0 op status=0 http_status=403 ======
2016-11-16 17:43:15.676071 7ffa95fdb700 20 process_request() returned -2027
2016-11-16 17:43:15.676215 7ffa95fdb700  1 civetweb: 0x7ffb04004120: 10.70.1.24 - - [16/Nov/2016:17:43:15 -0800] "PUT /test/ HTTP/1.0" 403 0 - Cyberduck/5.2.3.21496 (Windows 7/6.1) (x86)
Actions
Copy link
 #8

Updated by Robin Johnson over 6 years ago

  • Severity changed from 3 - minor to 2 - major
  • Release set to master
  • Release set to giant
  • Release set to kraken
  • Release set to luminous

I just hit this bug, and confirm it's still present in 10.2.9.

It's only occuring because of the awsv4 querystring auth, when a load balancer is being used such that RGW is not running on 80/443, but the client thinks it is.

The client-side logic:
It's on port 80/443, I don't need to add the port into the host token for hashing.
The server-side logic:
I'm running on a port other than 80/443, I need to add the port to the host token for hashing.

Then the have a mismatch, and it we have this bug.

https://github.com/ceph/ceph/commit/e9917a66d42b1900c0ce73f6748e9bc5187d8cd0 jmunhoz: first introduction
https://github.com/ceph/ceph/commit/033888bbd0e4d8d81358bf61a099276dddb5692b yehuda: don't add port for 80/443

I think that it should never have been added in the first place.
If the port is indeed not standard, it should to be included in the host header, per the HTTP standard.

Actions
Copy link
 #9

Updated by Robin Johnson over 6 years ago

  • Release deleted (master)
  • Release deleted (kraken)
  • Release deleted (luminous)
Actions
Copy link
 #10

Updated by Robin Johnson over 6 years ago

  • Backport set to jewel
Actions
Copy link
 #12

Updated by Nathan Cutler over 6 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #13

Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #20992: jewel: AWS4 Presigned URL not accepted by RGW added
Actions
Copy link
 #14

Updated by Nathan Cutler over 6 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF