Project

General

Profile

Bug #27217

rgw: Swift interface: server side copy fails if object name contains `?`

Added by Nick Craig-Wood over 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
luminous mimic nautilus
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Here is a demonstration of the problem with rclone. This sequence used to work with previous versions of CEPH and the same code works with openstack swift.

Create the object with a ? in hello?hello.txt

$ echo "hello" | rclone rcat 'swift:container1/hello?hello.txt'

Check it is OK

$ rclone ls swift:container1
        6 hello?hello.txt

Do a server side copy of hello?hello.txt to hello2?hello2.txt

rclone copyto -vv --dump bodies 'swift:container1/hello?hello.txt' 'swift:container1/hello2?hello2.txt'
[snip - see below for HTTP transaction]
2018/08/23 16:11:46 ERROR : hello?hello.txt: Failed to copy: object not found

See what happened in the container

$ rclone ls swift:container1
        6 hello2
        6 hello?hello.txt

So ceph/rgw truncated the file name of hello2?hello2.txt at the ? to get hello2.

Here is the COPY http transaction

2018/08/23 16:11:46 DEBUG : HTTP REQUEST (req 0xc000198f00)
2018/08/23 16:11:46 DEBUG : COPY /swift/v1/659259d53fc74389b9d71602f8a59bf8/container1/hello%3Fhello.txt HTTP/1.1
Host: storage.reading-a.openstack.memset.com:8080
User-Agent: rclone/v1.42-126-g577fcd31-fix-2464-swift-auth
Destination: container1/hello2?hello2.txt
X-Auth-Token: XXXX
Accept-Encoding: gzip

2018/08/23 16:11:46 DEBUG : HTTP RESPONSE (req 0xc000198f00)
2018/08/23 16:11:46 DEBUG : HTTP/1.1 201 Created
Access-Control-Allow-Headers: Origin,Content-Type,Cache-Control,Content-Language,Expires,Last-Modified,Pragma,X-Requested-With,User-Agent,If-Modified-Since,Accept,X-Auth-Token
Access-Control-Max-Age: 10800
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Fcontainer1
Content-Type: text/plain; charset=utf-8
Date: Thu, 23 Aug 2018 15:11:46 GMT
Etag: b1946ac92492d2347c6235b4d2611184
Last-Modified: Thu, 23 Aug 2018 15:11:46 GMT
X-Copied-From: container1/hello%3Fhello.txt
X-Copied-From-Account: 659259d53fc74389b9d71602f8a59bf8
X-Copied-From-Last-Modified: Thu, 23 Aug 2018 15:10:19 GMT
X-Object-Meta-Mtime: 1535037019.524649738
X-Openstack-Request-Id: tx000000000000000007c89-005b7eceb2-236cc2a-reading-a
X-Trans-Id: tx000000000000000007c89-005b7eceb2-236cc2a-reading-a
Content-Length: 0

Rclone writes the destination header as Destination: container1/hello2?hello2.txt

In https://developer.openstack.org/api-ref/object-store/ it states under the COPY method

Destination header string The container and object name of the destination object in the form of /container/object. You must UTF-8-encode and then URL-encode the names of the destination container and object before you include them in this header.

rclone is not URL encoding the ? in the Destination which is out of spec. However this used to work with previous versions of CEPH and works with OpenStack swift also.

If I change rclone to URL encode the Destination: header I get exactly the same results though:

2018/08/24 10:06:47 DEBUG : HTTP REQUEST (req 0xc0000f6300)
2018/08/24 10:06:47 DEBUG : COPY /swift/v1/659259d53fc74389b9d71602f8a59bf8/container1/hello%3Fhello.txt HTTP/1.1
Host: storage.reading-a.openstack.memset.com:8080
User-Agent: rclone/v1.42-DEV
Destination: container1/hello2%3Fhello2.txt
X-Auth-Token: XXXX
Accept-Encoding: gzip

2018/08/24 10:06:47 DEBUG : HTTP RESPONSE (req 0xc0000f6300)
2018/08/24 10:06:47 DEBUG : HTTP/1.1 201 Created
Access-Control-Allow-Headers: Origin,Content-Type,Cache-Control,Content-Language,Expires,Last-Modified,Pragma,X-Requested-With,User-Agent,If-Modified-Since,Accept,X-Auth-Token
Access-Control-Max-Age: 10800
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Fcontainer1
Content-Type: text/plain; charset=utf-8
Date: Fri, 24 Aug 2018 09:06:47 GMT
Etag: b1946ac92492d2347c6235b4d2611184
Last-Modified: Fri, 24 Aug 2018 09:06:47 GMT
X-Copied-From: container1/hello%3Fhello.txt
X-Copied-From-Account: 659259d53fc74389b9d71602f8a59bf8
X-Copied-From-Last-Modified: Thu, 23 Aug 2018 15:10:19 GMT
X-Object-Meta-Mtime: 1535037019.524649738
X-Openstack-Request-Id: tx0000000000000000376b0-005b7fcaa7-236cc2a-reading-a
X-Trans-Id: tx0000000000000000376b0-005b7fcaa7-236cc2a-reading-a
Content-Length: 0

The result is the same with CEPH apparently ignoring everything after the URL encoded ? and truncating the name to just hello2.

$ rclone ls swift:container1
        6 hello2
        6 hello?hello.txt

We've encountered this on our Ceph Luminous 12.2.7 Cluster after an upgrade from Kraken.


Related issues

Copied to rgw - Backport #40127: luminous: rgw: Swift interface: server side copy fails if object name contains `?` Resolved
Copied to rgw - Backport #40128: mimic: rgw: Swift interface: server side copy fails if object name contains `?` Resolved
Copied to rgw - Backport #40129: nautilus: rgw: Swift interface: server side copy fails if object name contains `?` Resolved

History

#1 Updated by John Spray over 5 years ago

  • Project changed from Ceph to rgw
  • Category deleted (openstack)

#2 Updated by Casey Bodley over 5 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Casey Bodley
  • Backport set to luminous mimic

#3 Updated by Nick Craig-Wood about 5 years ago

I just notices that this exact same bug reproduces on the S3 interface too, the relevant transaction being:

2019/01/09 10:44:00 DEBUG : HTTP REQUEST (req 0xc00048c400)
2019/01/09 10:44:00 DEBUG : PUT /container1/hello2%3Fhello2.txt HTTP/1.1
Host: storage.dunsfold-a.openstack.memset.com:8080
User-Agent: rclone/v1.45-069-g7ee7bc87-beta
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: 
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Copy-Source: container1/hello%3Fhello.txt
X-Amz-Date: 20190109T104400Z
X-Amz-Metadata-Directive: COPY
Accept-Encoding: gzip

2019/01/09 10:44:00 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/01/09 10:44:00 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/01/09 10:44:00 DEBUG : HTTP RESPONSE (req 0xc00048c400)
2019/01/09 10:44:00 DEBUG : HTTP/1.1 404 Not Found
Content-Length: 232
Accept-Ranges: bytes
Access-Control-Allow-Headers: Origin,Content-Type,Cache-Control,Content-Language,Expires,Last-Modified,Pragma,X-Requested-With,User-Agent,If-Modified-Since,Accept,X-Auth-Token
Access-Control-Max-Age: 10800
Bucket: 659259d53fc74389b9d71602f8a59bf8%2Fcontainer1
Content-Type: application/xml
Date: Wed, 09 Jan 2019 10:44:00 GMT
X-Amz-Request-Id: tx00000000000000081eff9-005c35d070-3449ba4-dunsfold-a

<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>container1</BucketName><RequestId>tx00000000000000081eff9-005c35d070-3449ba4-dunsfold-a</RequestId><HostId>3449ba4-dunsfold-a-dunsfold-a</HostId></Error>

#4 Updated by Casey Bodley almost 5 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from luminous mimic to luminous mimic nautilus

#5 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #40127: luminous: rgw: Swift interface: server side copy fails if object name contains `?` added

#6 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #40128: mimic: rgw: Swift interface: server side copy fails if object name contains `?` added

#7 Updated by Nathan Cutler almost 5 years ago

  • Copied to Backport #40129: nautilus: rgw: Swift interface: server side copy fails if object name contains `?` added

#8 Updated by Nathan Cutler almost 5 years ago

  • Pull request ID set to 25498

#9 Updated by Nathan Cutler over 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF