Ceph » rgw

@PL Garnier, could you share the version(s) where you've reproduced this?

thanks!

Matt

I couldn't find how to edit the issue to add that info in the ticket.

I can reproduce with Luminous (including v12.2.2), I haven't tried with earlier versions.

For me, testing with recent-ish master, it looks like boto3 trivially doesn't like the output from the first create bucket when run as is:

Traceback (most recent call last):
File "./copy-mp-err.py", line 33, in <module>
s3client.create_bucket(Bucket=BUCKET)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/client.py", line 312, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/client.py", line 592, in _make_api_call
operation_model, request_dict)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/endpoint.py", line 168, in _send_request
request, operation_model, attempts)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/endpoint.py", line 233, in _get_response
response_dict, operation_model.output_shape)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/parsers.py", line 212, in parse
parsed = self._do_parse(response, shape)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/parsers.py", line 614, in _do_parse
self._parse_payload(response, shape, member_shapes, final_parsed)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/parsers.py", line 647, in _parse_payload
original_parsed = self._initial_body_parse(response['body'])
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/parsers.py", line 729, in _initial_body_parse
return self._parse_xml_string_to_dom(xml_string)
File "/usr/lib/python2.7/site-packages/botocore-1.7.36-py2.7.egg/botocore/parsers.py", line 390, in _parse_xml_string_to_dom
"invalid XML received:\n%s" % (e, xml_string))
botocore.parsers.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received: {"entry_point_object_ver":{"tag":"_vmmatjKo_RVIzhzEfdjZnWL","ver":1},"object_ver":{"tag":"_dhSLmmLcoifeDRi96LGe9n3","ver":1},"bucket_info":{"bucket":{"name":"COPY-MULTIPART-ERROR","marker":"733f69ca-7f7a-478a-9313-d8a33fabaf50.14105.1","bucket_id":"733f69ca-7f7a-478a-9313-d8a33fabaf50.14105.1","tenant":"","explicit_placement":{"data_pool":"","data_extra_pool":"","index_pool":""}},"creation_time":"2017-12-05 14:01:19.658737Z","owner":"testuser","flags":0,"zonegroup":"49a1e9d1-ed53-4318-aac9-719477ae9376","placement_rule":"default-placement","has_instance_obj":"true","quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"num_shards":0,"bi_shard_hash_type":0,"requester_pays":"false","has_website":"false","swift_versioning":"false","swift_ver_location":"","index_type":0,"mdsearch_config":[],"reshard_status":0,"new_bucket_instance_id":""}}

Sorry, that was just me being dumb, having turned my test user into a system-user:

Creating bucket s3://mylittlepony
Uploading s3://mylittlepony/src
Removing s3://mylittlepony/dst
Copying s3://mylittlepony/src to s3://mylittlepony/dst

Here's what I get with:

$ radosgw --version
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)

And boto3==1.4.8:

Traceback (most recent call last):
  File "./copy-multipart-error.py", line 38, in <module>
    s3client.copy(copy_source, BUCKET, DST, Config=tx_config)
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/boto3/s3/inject.py", line 271, in copy
    return future.result()
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/s3transfer/copies.py", line 312, in _main
    UploadId=upload_id, PartNumber=part_number, **extra_args)
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/plg/Code/ceph-repro/lib/python3.5/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the UploadPartCopy operation: Unknown

However, I just finished building the Luminous branch, and this seems to be fixed. Tested with 092ea5174c069f060bcccc1fa044d5e67da3938c.

(cbodley notest that my failure is an interaction with multisite-sync's overloading of "system-user" to induce extra output--which presumbably wasn't being parsed by boto? he suggests we should protect clients from multipart extra data, maybe by triggering only on a special header)

cool