Project

General

Profile

Bug #22121

rgw: S3 interface: X-Amz-Copy-Source must be URL-encoded

Added by Nick Craig-Wood about 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
11/14/2017
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:

Description

According to the AWS docs on COPY: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html it states about X-Amz-Copy-Source "This string must be URL-encoded".

However URL encoding it does not seem to work with CEPH.

Here are some examples generated with rclone trying to copy an object called "complex character £100.txt" from one bucket to another.

2017/11/14 09:22:18 DEBUG : HTTP REQUEST (req 0xc420352100)
2017/11/14 09:22:18 DEBUG : PUT /utf8-test2/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.38-107-gc03f558d-vfs-rw
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test%2Fcomplex+character+%C2%A3100.txt
X-Amz-Date: 20171114T092218Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip

@2017/11/14 09:22:18 DEBUG : HTTP RESPONSE (req 0xc420352100)
2017/11/14 09:22:18 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Futf8-test2
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:22:18 GMT
X-Amz-Request-Id: tx0000000000000005bdf86-005a0ab5ca-205796c-dunsfold-a

<Error><Code>NoSuchKey</Code><BucketName>utf8-test2</BucketName><RequestId>tx0000000000000005bdf86-005a0ab5ca-205796c-dunsfold-a</RequestId><HostId>205796c-dunsfold-a-dunsfold-a</HostId></Error>
@

However if I don't URL encode X-Amz-Copy-Source it does work

2017/11/14 09:38:06 DEBUG : HTTP REQUEST (req 0xc420128900)
2017/11/14 09:38:06 DEBUG : PUT /utf8-test3/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.38-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test/complex character £100.txt
X-Amz-Date: 20171114T093806Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip

@2017/11/14 09:38:06 DEBUG : HTTP RESPONSE (req 0xc420128900)
2017/11/14 09:38:06 DEBUG : HTTP/1.1 200 OK
Content-Length: 183
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Futf8-test3
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:38:06 GMT
X-Amz-Request-Id: tx0000000000000005bed5c-005a0ab97e-2027f9e-dunsfold-a

<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LastModified>2017-11-14T09:38:06.634Z</LastModified><ETag>b1946ac92492d2347c6235b4d2611184</ETag></CopyObjectResult>
@

However when trying with AWS the first of these works and the second doesn't. Here is the failing request from AWS

2017/11/14 09:40:53 DEBUG : HTTP REQUEST (req 0xc4202f4c00)
2017/11/14 09:40:53 DEBUG : PUT /utf8-test4/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: s3.amazonaws.com
User-Agent: rclone/v1.38-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test/complex character £100.txt
X-Amz-Date: 20171114T094053Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip

@2017/11/14 09:40:53 DEBUG : HTTP RESPONSE (req 0xc4202f4c00)
2017/11/14 09:40:53 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:40:53 GMT
Server: AmazonS3
X-Amz-Id-2: 0JisWvRlBmOU7tZrndAcVTenG81tdAj2EUCrfeFEKXdUDVONLI5QRFIEgQLg6roGBHa8QTZhJUA=
X-Amz-Request-Id: D57DF1E9FFE93CAC

103
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>complex character ??100.txt</Key><RequestId>D57DF1E9FFE93CAC</RequestId><HostId>0JisWvRlBmOU7tZrndAcVTenG81tdAj2EUCrfeFEKXdUDVONLI5QRFIEgQLg6roGBHa8QTZhJUA=</HostId></Error>
0
@

This leads me to believe that this is a deviation from the S3 spec by CEPH and hence a bug.

This bug affects rclone users in two ways, firstly that certain files can't be server side copied and secondly that these files can't have their metadata changed so the users can't update their modification time (which is held as a metadata key).

Originally from: https://github.com/ncw/rclone/issues/586


Related issues

Copied to rgw - Backport #22893: luminous: rgw: S3 interface: X-Amz-Copy-Source must be URL-encoded Resolved

History

#1 Updated by Matt Benjamin about 1 year ago

  • Status changed from New to Triaged
  • Assignee set to Matt Benjamin
  • Backport set to Luminous

#2 Updated by Matt Benjamin about 1 year ago

(we won't fix kraken, does this reproduce on Luminous or above?)

#3 Updated by Matt Benjamin about 1 year ago

  • Priority changed from Normal to High

#4 Updated by Matt Benjamin about 1 year ago

  • Status changed from Triaged to In Progress

#5 Updated by Nathan Cutler about 1 year ago

  • Backport changed from Luminous to luminous

#6 Updated by Matt Benjamin about 1 year ago

  • Status changed from In Progress to Need Review

#7 Updated by Matt Benjamin 12 months ago

  • Status changed from Need Review to Pending Backport

#8 Updated by Matt Benjamin 12 months ago

(luminous backport staged at https://github.com/ceph/ceph/pull/20236)

#9 Updated by Nathan Cutler 12 months ago

  • Copied to Backport #22893: luminous: rgw: S3 interface: X-Amz-Copy-Source must be URL-encoded added

#10 Updated by Nathan Cutler 9 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF