Bug #10015
closedrgw sync agent: 403 when syncing object that has tilde in its name
0%
Description
The cuplrit is the python requests module that the sync agent uses in order to send the http requets. Wittily the module decides that the tilde param should never be escaped and it unescapes it. This collidees with the liburl (that boto usese) escaping scheme that we use before signing the requests. One solution is to modify the requests library. Another solution would be to use different library for sending the requests, maybe something like this:
diff --git a/radosgw_agent/client.py b/radosgw_agent/client.py index 2b8f027..263bc1f 100644 --- a/radosgw_agent/client.py +++ b/radosgw_agent/client.py @@ -5,6 +5,7 @@ import logging import random import requests import urllib +import string from urlparse import urlparse from boto.connection import AWSAuthConnection @@ -111,7 +112,7 @@ def _build_request(conn, method, basepath='', resource = '', headers=None, conn, method, path, auth_path, params, headers, data, host) def check_result_status(result): - if result.status_code / 100 != 2: + if result.status / 100 != 2: raise code_to_exc.get(result.status_code, HttpError)(result.status_code, result.content) def url_safe(component): @@ -147,16 +148,17 @@ def request(connection, type_, resource, params=None, headers=None, request.authorize(connection=connection) - handler = getattr(requests, type_) boto.log.debug('url = %r\nparams=%r\nheaders=%r\ndata=%r', url, params, request.headers, data) - result = handler(url, params=params, headers=request.headers, data=data) + + result = connection.make_request(request.method, request.path, request.body, request.headers) check_result_status(result) if data or not expect_json: - return result.raw - return result.json() + return result + + return json.loads(result.read())
Updated by Yehuda Sadeh over 9 years ago
- Assignee changed from Josh Durgin to Alfredo Deza
Updated by Alfredo Deza over 9 years ago
- Status changed from New to In Progress
Confirmed that requests is doing the unquoting of a quoted url with the tilde char:
In [26]: print requests.get('http://127.0.0.1:8888/?tilde=%7E~').text {"tilde": "~~"}
That is running a local http server that returns the arguments it receives, I encoded the `~` char and it got 'translated' by `requests`.
Did some investigation and although there are patches available I still believe we should use boto here:
Updated by Alfredo Deza over 9 years ago
- Status changed from In Progress to Fix Under Review
- Priority changed from High to Normal
Updated by Yehuda Sadeh over 9 years ago
- Assignee changed from Alfredo Deza to Josh Durgin
Updated by Yehuda Sadeh over 9 years ago
- Assignee changed from Josh Durgin to Alfredo Deza
Updated by Alfredo Deza over 9 years ago
- Status changed from Fix Under Review to Resolved
Changes included in https://github.com/ceph/radosgw-agent/pull/14
merged commit fbcc438 into master
Updated by Ken Dreyer about 9 years ago
Just a note that this fix shipped in the 1.2 release. https://github.com/ceph/radosgw-agent/blob/master/CHANGELOG.rst#12