Bug #41870
rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set
0%
Description
I made a bucket with a single object in on CEPH and s3 with the name "prefix with + plus/file.txt". I then issued a GET on the bucket with prefix to list the top level directory of the bucket. S3 returns the Key and the CommonPrefixes url encoded, whereas CEPH only encodes the Key.
Here is the XML returned. Note the URL encoding in the s3 response for CommonPrefixes and the lack of it in the CEPH response.
CEPH
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Tenant>659259d53fc74389b9d71602f8a59bf8</Tenant>
<Name>rclone-plus</Name>
<Prefix>
</Prefix>
<Marker>
</Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<EncodingType>url</EncodingType>
<CommonPrefixes>
<Prefix>prefix with + plus/</Prefix>
</CommonPrefixes>
</ListBucketResult>
s3
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>rclone-plus</Name>
<Prefix>
</Prefix>
<Marker>
</Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<EncodingType>url</EncodingType>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>prefix+with+%2B+plus/</Prefix>
</CommonPrefixes>
</ListBucketResult>
And here are the full HTTP request and response
2019/09/16 13:44:27 DEBUG : GET /rclone-plus?delimiter=%2F&encoding-type=url&max-keys=1000&prefix= HTTP/1.1 Host: s3.eu-west-2.amazonaws.com User-Agent: rclone/v1.49.3-DEV Authorization: XXXX X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20190916T124427Z Accept-Encoding: gzip 2019/09/16 13:44:27 DEBUG : HTTP RESPONSE (req 0xc000578f00) 2019/09/16 13:44:27 DEBUG : HTTP/1.1 200 OK Transfer-Encoding: chunked Content-Type: application/xml Date: Mon, 16 Sep 2019 12:44:28 GMT Server: AmazonS3 X-Amz-Bucket-Region: eu-west-2 X-Amz-Id-2: ROeSmZNyUlxLSF1CYXlWlPkd54bUwD3wi99ziPKPuGbRJO1xrahGVcMTTMUvOLXi6pRvJ22k1zs= X-Amz-Request-Id: 2887BDCA1729272E 16c <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rclone-plus</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>prefix+with+%2B+plus/</Prefix></CommonPrefixes></ListBucketResult> 0
CEPH
2019/09/16 13:43:58 DEBUG : GET /rclone-plus?delimiter=%2F&encoding-type=url&max-keys=1000&prefix= HTTP/1.1 Host: storage.dunsfold-a.openstack.memset.com:8080 User-Agent: rclone/v1.49.3-DEV Authorization: XXXX X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20190916T124358Z Accept-Encoding: gzip 2019/09/16 13:43:58 DEBUG : HTTP RESPONSE (req 0xc000583500) 2019/09/16 13:43:58 DEBUG : HTTP/1.1 200 OK Content-Length: 410 Access-Control-Allow-Headers: Origin,Content-Type,Cache-Control,Content-Language,Expires,Last-Modified,Pragma,X-Requested-With,User-Agent,If-Modified-Since,Accept,X-Auth-Token Access-Control-Max-Age: 10800 Bucket: 659259d53fc74389b9d71602f8a59bf8%2Frclone-plus Content-Type: application/xml Date: Mon, 16 Sep 2019 12:43:58 GMT X-Amz-Request-Id: tx000000000000001f6a4ba-005d7f838e-3fdcb85-dunsfold-a <?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Tenant>659259d53fc74389b9d71602f8a59bf8</Tenant><Name>rclone-plus</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><EncodingType>url</EncodingType><CommonPrefixes><Prefix>prefix with + plus/</Prefix></CommonPrefixes></ListBucketResult>
I tested this with 12.2.12, luminous as it is the only platform I have easy access to. However looking at the source code I think this bug is still present in master.
Related issues
History
#1 Updated by Abhishek Lekshmanan over 4 years ago
- Assignee set to Abhishek Lekshmanan
#2 Updated by Abhishek Lekshmanan over 4 years ago
- Status changed from New to Triaged
#3 Updated by Abhishek Lekshmanan over 4 years ago
- Status changed from Triaged to Fix Under Review
- Pull request ID set to 30970
#4 Updated by Casey Bodley over 4 years ago
- Status changed from Fix Under Review to 7
#5 Updated by Casey Bodley over 4 years ago
test case in https://github.com/ceph/s3-tests/pull/311
#6 Updated by J. Eric Ivancich over 4 years ago
- Target version set to v15.0.0
#7 Updated by Abhishek Lekshmanan over 4 years ago
- Status changed from 7 to Pending Backport
#8 Updated by Abhishek Lekshmanan over 4 years ago
- Backport set to nautilus, mimic
#9 Updated by Nathan Cutler over 4 years ago
- Copied to Backport #43160: mimic: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set added
#10 Updated by Nathan Cutler over 4 years ago
- Copied to Backport #43161: nautilus: rgw: s3 interface doesn't URL encode CommonPrefixes when URL encoding is set added
#11 Updated by Nathan Cutler about 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".