Project

General

Profile

Bug #41870

rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set

Added by Nick Craig-Wood over 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus, mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I made a bucket with a single object in on CEPH and s3 with the name "prefix with + plus/file.txt". I then issued a GET on the bucket with prefix to list the top level directory of the bucket. S3 returns the Key and the CommonPrefixes url encoded, whereas CEPH only encodes the Key.

Here is the XML returned. Note the URL encoding in the s3 response for CommonPrefixes and the lack of it in the CEPH response.

CEPH

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Tenant>659259d53fc74389b9d71602f8a59bf8</Tenant>
 <Name>rclone-plus</Name>
 <Prefix>
 </Prefix>
 <Marker>
 </Marker>
 <MaxKeys>1000</MaxKeys>
 <Delimiter>/</Delimiter>
 <IsTruncated>false</IsTruncated>
 <EncodingType>url</EncodingType>
 <CommonPrefixes>
  <Prefix>prefix with + plus/</Prefix>
 </CommonPrefixes>
</ListBucketResult>

s3

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Name>rclone-plus</Name>
 <Prefix>
 </Prefix>
 <Marker>
 </Marker>
 <MaxKeys>1000</MaxKeys>
 <Delimiter>/</Delimiter>
 <EncodingType>url</EncodingType>
 <IsTruncated>false</IsTruncated>
 <CommonPrefixes>
  <Prefix>prefix+with+%2B+plus/</Prefix>
 </CommonPrefixes>
</ListBucketResult>

And here are the full HTTP request and response

2019/09/16 13:44:27 DEBUG : GET /rclone-plus?delimiter=%2F&encoding-type=url&max-keys=1000&prefix= HTTP/1.1
Host: s3.eu-west-2.amazonaws.com
User-Agent: rclone/v1.49.3-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190916T124427Z
Accept-Encoding: gzip

2019/09/16 13:44:27 DEBUG : HTTP RESPONSE (req 0xc000578f00)
2019/09/16 13:44:27 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 16 Sep 2019 12:44:28 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-west-2
X-Amz-Id-2: ROeSmZNyUlxLSF1CYXlWlPkd54bUwD3wi99ziPKPuGbRJO1xrahGVcMTTMUvOLXi6pRvJ22k1zs=
X-Amz-Request-Id: 2887BDCA1729272E

16c
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rclone-plus</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>prefix+with+%2B+plus/</Prefix></CommonPrefixes></ListBucketResult>
0

CEPH

2019/09/16 13:43:58 DEBUG : GET /rclone-plus?delimiter=%2F&encoding-type=url&max-keys=1000&prefix= HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.49.3-DEV
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20190916T124358Z
Accept-Encoding: gzip

2019/09/16 13:43:58 DEBUG : HTTP RESPONSE (req 0xc000583500)
2019/09/16 13:43:58 DEBUG : HTTP/1.1 200 OK
Content-Length: 410
Access-Control-Allow-Headers: Origin,Content-Type,Cache-Control,Content-Language,Expires,Last-Modified,Pragma,X-Requested-With,User-Agent,If-Modified-Since,Accept,X-Auth-Token
Access-Control-Max-Age: 10800
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Frclone-plus
Content-Type: application/xml
Date: Mon, 16 Sep 2019 12:43:58 GMT
X-Amz-Request-Id: tx000000000000001f6a4ba-005d7f838e-3fdcb85-dunsfold-a

<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Tenant>659259d53fc74389b9d71602f8a59bf8</Tenant><Name>rclone-plus</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><EncodingType>url</EncodingType><CommonPrefixes><Prefix>prefix with + plus/</Prefix></CommonPrefixes></ListBucketResult>

I tested this with 12.2.12, luminous as it is the only platform I have easy access to. However looking at the source code I think this bug is still present in master.


Related issues

Copied to rgw - Backport #43160: mimic: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set Rejected
Copied to rgw - Backport #43161: nautilus: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set Resolved

History

#1 Updated by Abhishek Lekshmanan over 4 years ago

  • Assignee set to Abhishek Lekshmanan

#2 Updated by Abhishek Lekshmanan over 4 years ago

  • Status changed from New to Triaged

#3 Updated by Abhishek Lekshmanan over 4 years ago

  • Status changed from Triaged to Fix Under Review
  • Pull request ID set to 30970

#4 Updated by Casey Bodley over 4 years ago

  • Status changed from Fix Under Review to 7

#6 Updated by J. Eric Ivancich over 4 years ago

  • Target version set to v15.0.0

#7 Updated by Abhishek Lekshmanan over 4 years ago

  • Status changed from 7 to Pending Backport

#8 Updated by Abhishek Lekshmanan over 4 years ago

  • Backport set to nautilus, mimic

#9 Updated by Nathan Cutler over 4 years ago

  • Copied to Backport #43160: mimic: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set added

#10 Updated by Nathan Cutler over 4 years ago

  • Copied to Backport #43161: nautilus: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set added

#11 Updated by Nathan Cutler about 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF