Ceph » rgw

radosgw --version
Version: ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)

How to reproduce:

Create a user "testtenant$testuser" (testtenant tenant), and a bucket owned by that user
"testtenant/copybucket" which will be referred to as "copybucket" for those
credentials.

dd if=/dev/zero bs=1M count=30 of=dummydata
aws s3 --endpoint-url "..." cp dummydata s3://copybucket/dummydata
aws s3 --endpoint-url "..." mv s3://copybucket/dummydata s3://copybucket/dummydata2

You will get error messages in the form:

move failed: s3://copybucket/dummydata to s3://copybucket/dummydata2 An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

From the radosgw logs:

2020-04-08 11:18:41.980 7f7ef4720700  1 ====== starting new request req=0x7f7ef47198c0 =====
2020-04-08 11:18:41.984 7f7ef4720700  1 ====== req done req=0x7f7ef47198c0 op status=0 http_status=200 latency=0.00399998s ======
2020-04-08 11:18:41.984 7f7ef4720700  1 civetweb: 0x565436716000: 127.0.0.1 - - [08/Apr/2020:11:18:41 +0200] "HEAD /copybucket/dummydata HTTP/1.1" 200 325 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.116 7f7ef3f1f700  1 ====== starting new request req=0x7f7ef3f188c0 =====
2020-04-08 11:18:42.128 7f7ef3f1f700  1 ====== req done req=0x7f7ef3f188c0 op status=0 http_status=200 latency=0.0119999s ======
2020-04-08 11:18:42.128 7f7ef3f1f700  1 civetweb: 0x5654367169d8: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "POST /copybucket/dummydata2?uploads HTTP/1.1" 200 457 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.152 7f7ef3f1f700  1 ====== starting new request req=0x7f7ef3f188c0 =====
2020-04-08 11:18:42.156 7f7ef3f1f700  1 ====== req done req=0x7f7ef3f188c0 op status=0 http_status=404 latency=0.00399998s ======
2020-04-08 11:18:42.156 7f7ef3f1f700  1 civetweb: 0x5654367169d8: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=3 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef371e700  1 ====== starting new request req=0x7f7ef37178c0 =====
2020-04-08 11:18:42.320 7f7ef2f1d700  1 ====== starting new request req=0x7f7ef2f168c0 =====
2020-04-08 11:18:42.320 7f7ef371e700  1 ====== req done req=0x7f7ef37178c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef371e700  1 civetweb: 0x5654367173b0: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=1 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef2f1d700  1 ====== req done req=0x7f7ef2f168c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef2f1d700  1 civetweb: 0x565436717d88: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=4 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.320 7f7ef271c700  1 ====== starting new request req=0x7f7ef27158c0 =====
2020-04-08 11:18:42.320 7f7ef271c700  1 ====== req done req=0x7f7ef27158c0 op status=0 http_status=404 latency=0s ======
2020-04-08 11:18:42.320 7f7ef271c700  1 civetweb: 0x565436718760: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "PUT /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM&partNumber=2 HTTP/1.1" 404 431 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37
2020-04-08 11:18:42.360 7f7ef271c700  1 ====== starting new request req=0x7f7ef27158c0 =====
2020-04-08 11:18:42.372 7f7ef271c700  1 ====== req done req=0x7f7ef27158c0 op status=0 http_status=204 latency=0.0119999s ======
2020-04-08 11:18:42.372 7f7ef271c700  1 civetweb: 0x565436718760: 127.0.0.1 - - [08/Apr/2020:11:18:42 +0200] "DELETE /copybucket/dummydata2?uploadId=2~h50J-wCUBNz9xvz8B8S-uwmmA_ecoLM HTTP/1.1" 204 133 - aws-cli/1.18.37 Python/3.8.2 Linux/5.6.2-arch1-2 botocore/1.15.37

Alternative reproducer, use the copy-multipart-error.py script from
https://tracker.ceph.com/issues/22320.

Then you get a trace like so:

Creating bucket s3://copybucket
Uploading s3://copybucket/src
Removing s3://copybucket/dst
Copying s3://copybucket/src to s3://copybucket/dst
Traceback (most recent call last):
  File "copy-multipart-error.py", line 49, in <module>
    s3client.copy(copy_source, BUCKET, DST, Config=tx_config)
  File "/usr/lib/python3.8/site-packages/boto3/s3/inject.py", line 379, in copy
    return future.result()
  File "/usr/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3.8/site-packages/s3transfer/copies.py", line 318, in _main
    response = client.upload_part_copy(
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python3.8/site-packages/botocore/client.py", line 626, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

Work-around is to not use multitenant buckets and users.