Project

General

Profile

Bug #15896

S3: set EncodingType in ListBucketResult

Added by Loïc Dachary almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):


Related issues

Copied to rgw - Backport #15897: hammer: S3: set EncodingType in ListBucketResult Resolved

History

#1 Updated by Loïc Dachary almost 8 years ago

  • Copied to Backport #15897: hammer: S3: set EncodingType in ListBucketResult added

#2 Updated by Loïc Dachary over 7 years ago

  • Status changed from Pending Backport to Resolved

#3 Updated by Andy Yang over 7 years ago

Hi there,

We've been watching this ticket with interest (and https://github.com/ceph/ceph/pull/7712) because this issue appears to be the root cause behind a great deal of pain for us. (note: I would have severity to 'High')

We are currently running 0.94.9 and AWS S3 commands are still not behaving correctly.

For example:

$ aws s3 --endpoint-url https://object.myradosgw.org:9080 cp s3://my.bucket.src/data/ s3://my.bucket.dest/data/ --exclude "*" --include "eb08*" --recursive

We want to copy all the files in s3://my.bucket.src/data into s3://my.bucket.dest/data if the file begins with 'eb08'. However, the cp command fails with the following errors:

copy failed: s3://my.bucket.src/data%2F00b2a66a-8bad-58f3-8674-dca412d8b846 to s3://my.bucket.dest/data/2F00b2a66a-8bad-58f3-8674-dca412d8b846 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b13cae-8761-56db-bcdd-4030c36148be to s3://my.bucket.dest/data/2F00b13cae-8761-56db-bcdd-4030c36148be An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
copy failed: s3://my.bucket.src/data%2F00b1eebe-15b4-586d-a76a-a8701af83073 to s3://my.bucket.dest/data/2F00b1eebe-15b4-586d-a76a-a8701af83073 An error occurred (NoSuchKey) when calling the CopyObject operation: Unknown
<---8<---snip--->

The copy operation is not interpreting the prefix slash properly. While it's not clear whether the NoSuchKey is happening on the source or destination side, it is definitely not handling the target path construction properly. There is something odd in the source path handling as well, since the files that are being "copied" do not satisfy the --include condition. They are the first three files that occur in the container lexicographically (kind of - the command listed them out of order - because of multiple child threads?)

This command works as expected against regular AWS S3 buckets.

#4 Updated by Yehuda Sadeh over 7 years ago

@ayang, the fix for this has made it into 0.94.9, so what you're seeing could be a different issue. Can you provide a complete scenario where this fails?

#5 Updated by Andy Yang over 7 years ago

Well.

Our logs show that 0.94.9 was applied in our environment on August 31, 2016.

I created this ticket when my test cases failed around Sept 12, 2016.

Today, I tried to reproduce the problem and I couldn't.

I guess I will close the ticket. Hope there is an option for "Blame sunspots"

Thanks for looking into this!

Also available in: Atom PDF