Ceph » rgw

Hi there,

We've been watching [[http://tracker.ceph.com/issues/15896]] with interest because this issue appears to be the root cause behind a great deal of pain for us. (note: I would have severity to 'High')

We are currently running 0.94.9 and AWS S3 commands are still not behaving correctly.

For example:

$ aws s3 --endpoint-url https://object.myradosgw.org:9080 cp s3://my.bucket.src/data/ s3://my.bucket.dest/data/ --exclude "*" --include "eb08*" --recursive

We want to copy all the files in s3://my.bucket.src/data into s3://my.bucket.dest/data if the file begins with 'eb08'. However, the cp command fails with the following errors:

copy failed: s3://my.bucket.src/data%2F00b2a66a-8bad-58f3-8674-dca412d8b846 to s3://my.bucket.dest/data/2F00b2a66a-8bad-58f3-8674-dca412d8b846 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b13cae-8761-56db-bcdd-4030c36148be to s3://my.bucket.dest/data/2F00b13cae-8761-56db-bcdd-4030c36148be An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b1eebe-15b4-586d-a76a-a8701af83073 to s3://my.bucket.dest/data/2F00b1eebe-15b4-586d-a76a-a8701af83073 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
<---8<---snip--->

The copy operation is not interpreting the prefix slash properly. While it's not clear whether the NoSuchKey is happening on the source or destination side, it is definitely not handling the target path construction properly. There is something odd in the source path handling as well, since the files that are being "copied" do not satisfy the --include condition. They are the first three files that occur in the container lexicographically (kind of - the command listed them out of order - because of multiple child threads?)

The cp command works properly for a single file copy:

$ aws s3 --endpoint-url https://object.myradosgw.org:9080 cp s3://my.bucket.src/data/3cbc006a-966f-5a79-b384-241393f92454 s3://my.bucket.dest/data/

copy: s3://my.bucket.src/data/3cbc006a-966f-5a79-b384-241393f92454 to s3://my.bucket.dest/data/3cbc006a-966f-5a79-b384-241393f92454

The above command works as expected against regular AWS S3 buckets.