Project

General

Profile

Actions
Copy link

Bug #45033

open

S3 multipart copy broken by multi tenant

Added by Asbjørn Sannes about 4 years ago. Updated almost 3 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Abhishek Lekshmanan
Target version:
-
% Done:

0%

Source:
Tags:
rgw s3
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.8
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

radosgw --version
Version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)

How to reproduce:

Create a user "testtenant$testuser" (testtenant tenant), and a bucket owned by that user
"testtenant/copybucket" which will be referred to as "copybucket" for those
credentials.

dd if=/dev/zero bs=1M count=30 of=dummydata
aws s3 --endpoint-url "..." cp dummydata s3://copybucket/dummydata
aws s3 --endpoint-url "..." mv s3://copybucket/dummydata s3://copybucket/dummydata2

You will get error messages in the form:

move failed: s3://copybucket/dummydata to s3://copybucket/dummydata2 An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

From the radosgw logs:

2020-04-08 11:18:41.980 7f7ef4720700  1 ====== starting new request req=0x7f7ef47198c0 =====
2020-04-08 11:18:41.984 7f7ef4720700  1 ====== req done req=0x7f7ef47198c0 op status=0 http_status=200 latency=0.00399998s ======
2020-04-08 11:18:41.984 7f7ef4720700  1 civetweb: 0x565436716000: 127.0.0.1 - - [08/Apr/2020:11:18:41 +0200] "HEAD /copybucket/dummydata HTTP/1.1" 200 325 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.116 7f7ef3f1f700  1 ====== starting new request req=0x7f7ef3f188c0 =====
2020-04-08 11:18:42.128 7f7ef3f1f700  1 ====== req done req=0x7f7ef3f188c0 op status=0 http_status=200 latency=0.0119999s ======
2020-04-08 11:18:42.128 7f7ef3f1f700  1 civetweb: 0x5654367169d8: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "POST /copybucket/dummydata2?uploads HTTP/1.1" 200 457 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.152 7f7ef3f1f700  1 ====== starting new request req=0x7f7ef3f188c0 =====
2020-04-08 11:18:42.156 7f7ef3f1f700  1 ====== req done req=0x7f7ef3f188c0 op status=0 http_status=404 latency=0.00399998s ======
2020-04-08 11:18:42.156 7f7ef3f1f700  1 civetweb: 0x5654367169d8: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=3 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef371e700  1 ====== starting new request req=0x7f7ef37178c0 =====
2020-04-08 11:18:42.320 7f7ef2f1d700  1 ====== starting new request req=0x7f7ef2f168c0 =====
2020-04-08 11:18:42.320 7f7ef371e700  1 ====== req done req=0x7f7ef37178c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef371e700  1 civetweb: 0x5654367173b0: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=1 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef2f1d700  1 ====== req done req=0x7f7ef2f168c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef2f1d700  1 civetweb: 0x565436717d88: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=4 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef271c700  1 ====== starting new request req=0x7f7ef27158c0 =====
2020-04-08 11:18:42.320 7f7ef271c700  1 ====== req done req=0x7f7ef27158c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef271c700  1 civetweb: 0x565436718760: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=2 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.360 7f7ef271c700  1 ====== starting new request req=0x7f7ef27158c0 =====
2020-04-08 11:18:42.372 7f7ef271c700  1 ====== req done req=0x7f7ef27158c0 op status=0 http_status=204 latency=0.0119999s ======
2020-04-08 11:18:42.372 7f7ef271c700  1 civetweb: 0x565436718760: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "DELETE /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM HTTP/1.1" 204 133 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37

Alternative reproducer, use the copy-multipart-error.py script from
https://tracker.ceph.com/issues/22320.

Then you get a trace like so:

Creating bucket s3://copybucket
Uploading s3://copybucket/src
Removing s3://copybucket/dst
Copying s3://copybucket/src to s3://copybucket/dst
Traceback (most recent call last):
  File "copy-multipart-error.py", line 49, in <module>
    s3client.copy(copy_source, BUCKET, DST, Config=tx_config)
  File "/usr/lib/python3.8/site-packages/boto3/s3/inject.py", line 379, in copy
    return future.result()
  File "/usr/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3.8/site-packages/s3transfer/copies.py", line 318, in _main
    response = client.upload_part_copy(
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

Work-around is to not use multitenant buckets and users.

Actions
Copy link
 #1

Updated by Abhishek Lekshmanan about 4 years ago

  • Assignee set to Abhishek Lekshmanan
Actions
Copy link
 #2

Updated by Abhishek Lekshmanan almost 4 years ago

For multipart copy and tenanted buckets, you need to use the tenant:bucket as the source bucket, otherwise the bucket being referred to falls back the bucket on non tenanted namespace, can you see if that format works (may need to adjust tooling, this was tested with aws cli and boto3), see eg. below where a bucket bucketc/mpc0 is a copy-part from a bucket tenantb/key1 of tenant testx

 aws s3api upload-part-copy --bucket bucketc --key mpc0 --part-number 0 --upload-id 2~0cPRSl5o5bkt3rG2hmuwIfE_NwsWKan --endpoint=http://localhost:8000 --copy-source testx:tenantb/key1
Actions
Copy link
 #3

Updated by Abhishek Lekshmanan almost 4 years ago

  • Status changed from New to Need More Info
Actions
Copy link
 #4

Updated by Asbjørn Sannes almost 4 years ago

If I try to use the tentant in the bucket name then I get an error (both when creating a bucket and when uploading):
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "testtenant$copybucket": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

Since the $ is is not allowed in the name.

It also kind of defeats the purpose of tenants having their own namespace?
I would expect it to behave in the same way as when you create a bucket, which now creates the bucket in the tenant namespace.

Actions
Copy link
 #5

Updated by Matt Benjamin almost 4 years ago

Asbjørn Sannes wrote:

If I try to use the tentant in the bucket name then I get an error (both when creating a bucket and when uploading):
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "testtenant$copybucket": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

This is internal to the boto3 client. We have open issues w/the package regarding this. You might need to patch if you wish to use it for this action.

Since the $ is is not allowed in the name.

It also kind of defeats the purpose of tenants having their own namespace?
I would expect it to behave in the same way as when you create a bucket, which now creates the bucket in the tenant namespace.

I think it's just a bug.

Matt

Actions
Copy link
 #6

Updated by Abhishek Lekshmanan almost 4 years ago

  • Status changed from Need More Info to In Progress

Asbjørn Sannes wrote:

If I try to use the tentant in the bucket name then I get an error (both when creating a bucket and when uploading):
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "testtenant$copybucket": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

Since the $ is is not allowed in the name.

It also kind of defeats the purpose of tenants having their own namespace?
I would expect it to behave in the same way as when you create a bucket, which now creates the bucket in the tenant namespace.

Actions
Copy link
 #7

Updated by Asbjørn Sannes over 3 years ago

Still a problem on 14.2.15.

Matt: I just realized I might have misinterpreted you answer, just to clarify, you believe it is a bug when:
1. multipart copy seems to not take into account the tenant namespace for the target?
or:
2. when you create a bucket it creates it in the tenant namespace?
3. that the boto library should allow $ in the bucket name?

Actions
Copy link
 #8

Updated by Asbjørn Sannes almost 3 years ago

Seems to work without issue on 16.2.4, close this then?

Actions
Copy link

Also available in: Atom PDF