Bug #15896
closedS3: set EncodingType in ListBucketResult
0%
Updated by Loïc Dachary almost 8 years ago
- Copied to Backport #15897: hammer: S3: set EncodingType in ListBucketResult added
Updated by Loïc Dachary over 7 years ago
- Status changed from Pending Backport to Resolved
Updated by Andy Yang over 7 years ago
Hi there,
We've been watching this ticket with interest (and https://github.com/ceph/ceph/pull/7712) because this issue appears to be the root cause behind a great deal of pain for us. (note: I would have severity to 'High')
We are currently running 0.94.9 and AWS S3 commands are still not behaving correctly.
For example:
$ aws s3 --endpoint-url https://object.myradosgw.org:9080 cp s3://my.bucket.src/data/ s3://my.bucket.dest/data/ --exclude "*" --include "eb08*" --recursive
We want to copy all the files in s3://my.bucket.src/data into s3://my.bucket.dest/data if the file begins with 'eb08'. However, the cp command fails with the following errors:
copy failed: s3://my.bucket.src/data%2F00b2a66a-8bad-58f3-8674-dca412d8b846 to s3://my.bucket.dest/data/2F00b2a66a-8bad-58f3-8674-dca412d8b846 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b13cae-8761-56db-bcdd-4030c36148be to s3://my.bucket.dest/data/2F00b13cae-8761-56db-bcdd-4030c36148be An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b1eebe-15b4-586d-a76a-a8701af83073 to s3://my.bucket.dest/data/2F00b1eebe-15b4-586d-a76a-a8701af83073 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
<---8<---snip--->
The copy operation is not interpreting the prefix slash properly. While it's not clear whether the NoSuchKey is happening on the source or destination side, it is definitely not handling the target path construction properly. There is something odd in the source path handling as well, since the files that are being "copied" do not satisfy the --include condition. They are the first three files that occur in the container lexicographically (kind of - the command listed them out of order - because of multiple child threads?)
This command works as expected against regular AWS S3 buckets.
Updated by Yehuda Sadeh over 7 years ago
@Andy Yang, the fix for this has made it into 0.94.9, so what you're seeing could be a different issue. Can you provide a complete scenario where this fails?
Updated by Andy Yang over 7 years ago
Well.
Our logs show that 0.94.9 was applied in our environment on August 31, 2016.
I created this ticket when my test cases failed around Sept 12, 2016.
Today, I tried to reproduce the problem and I couldn't.
I guess I will close the ticket. Hope there is an option for "Blame sunspots"
Thanks for looking into this!