Ceph » rgw

According to the AWS docs on COPY: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html it states about X-Amz-Copy-Source "This string must be URL-encoded".

However URL encoding it does not seem to work with CEPH.

Here are some examples generated with rclone trying to copy an object called "complex character £100.txt" from one bucket to another.

2017/11/14 09:22:18 DEBUG : HTTP REQUEST (req 0xc420352100)
2017/11/14 09:22:18 DEBUG : PUT /utf8-test2/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.38-107-gc03f558d-vfs-rw
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test%2Fcomplex+character+%C2%A3100.txt
X-Amz-Date: 20171114T092218Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip


@2017/11/14 09:22:18 DEBUG : HTTP RESPONSE (req 0xc420352100)
2017/11/14 09:22:18 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Futf8-test2
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:22:18 GMT
X-Amz-Request-Id: tx0000000000000005bdf86-005a0ab5ca-205796c-dunsfold-a

<Error><Code>NoSuchKey</Code><BucketName>utf8-test2</BucketName><RequestId>tx0000000000000005bdf86-005a0ab5ca-205796c-dunsfold-a</RequestId><HostId>205796c-dunsfold-a-dunsfold-a</HostId></Error>
@

However if I don't URL encode X-Amz-Copy-Source it does work

2017/11/14 09:38:06 DEBUG : HTTP REQUEST (req 0xc420128900)
2017/11/14 09:38:06 DEBUG : PUT /utf8-test3/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.38-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test/complex character £100.txt
X-Amz-Date: 20171114T093806Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip


@2017/11/14 09:38:06 DEBUG : HTTP RESPONSE (req 0xc420128900)
2017/11/14 09:38:06 DEBUG : HTTP/1.1 200 OK
Content-Length: 183
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Futf8-test3
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:38:06 GMT
X-Amz-Request-Id: tx0000000000000005bed5c-005a0ab97e-2027f9e-dunsfold-a

<CopyObjectResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/&quot;&gt;&lt;LastModified&gt;2017-11-14T09:38:06.634Z&lt;/LastModified&gt;&lt;ETag&gt;b1946ac92492d2347c6235b4d2611184&lt;/ETag&gt;&lt;/CopyObjectResult>
@

However when trying with AWS the first of these works and the second doesn't. Here is the failing request from AWS

2017/11/14 09:40:53 DEBUG : HTTP REQUEST (req 0xc4202f4c00)
2017/11/14 09:40:53 DEBUG : PUT /utf8-test4/complex%20character%20%C2%A3100.txt HTTP/1.1
Host: s3.amazonaws.com
User-Agent: rclone/v1.38-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: utf8-test/complex character £100.txt
X-Amz-Date: 20171114T094053Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip


@2017/11/14 09:40:53 DEBUG : HTTP RESPONSE (req 0xc4202f4c00)
2017/11/14 09:40:53 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 14 Nov 2017 09:40:53 GMT
Server: AmazonS3
X-Amz-Id-2: 0JisWvRlBmOU7tZrndAcVTenG81tdAj2EUCrfeFEKXdUDVONLI5QRFIEgQLg6roGBHa8QTZhJUA=
X-Amz-Request-Id: D57DF1E9FFE93CAC

103
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>complex character ??100.txt</Key><RequestId>D57DF1E9FFE93CAC</RequestId><HostId>0JisWvRlBmOU7tZrndAcVTenG81tdAj2EUCrfeFEKXdUDVONLI5QRFIEgQLg6roGBHa8QTZhJUA=</HostId></Error>
0
@

This leads me to believe that this is a deviation from the S3 spec by CEPH and hence a bug.

This bug affects rclone users in two ways, firstly that certain files can't be server side copied and secondly that these files can't have their metadata changed so the users can't update their modification time (which is held as a metadata key).

Originally from: https://github.com/ncw/rclone/issues/586