Project

General

Profile

Documentation #18889

rgw: S3 create bucket should not do response in json

Added by okwap okwap 6 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Target version:
-
Start date:
02/13/2017
Due date:
% Done:

0%

Tags:
Backport:
jewel,kraken
Reviewed:
Affected Versions:
Release:

Description

rgw s3 cannot work with boto3, due to create bucket returns json instead of xml.

boto3 error:

botocore.parsers.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received:
{"entry_point_object_ver":{"tag":"__3hTtMdsXlx-OJ9e_3RrkQC","ver":1},"object_ver":{"tag":"_jhpZpHMzyUsIvMSTqDws4T_","ver":1},"bucket_info":{"bucket":{"name":"python-sdk-sample-4b46707c-6408-4de8-b8a5-d3d8816ee5df","pool":".rgw.buckets","data_extra_pool":"","index_pool":".rgw.buckets","marker":"dev.6607669.7","bucket_id":"dev.6607669.7"},"creation_time":1486954351,"owner":"johndoe","flags":0,"region":"default","placement_rule":"","has_instance_obj":"true","quota":{"enabled":false,"max_size_kb":-1,"max_objects":-1},"num_shards":0,"bi_shard_hash_type":0}}
petertc@petertc-VirtualBox:~/aws-python-sample$ echo $no_proxy

packets:

T 10.0.2.15:43350 -> 10.144.225.175:8080 [AP]
PUT /my-new-bucket/ HTTP/1.1..Host: 10.144.225.175:8080..Accept-Encoding: identity..Date: Mon, 13 Feb 2017 03:15:09 GMT..Content-Length: 0..Authorization: AWS XS3SDAYG61ODT9K4Y6QL:QjC3I5C0s5SAs1J4Qv5
bziF+c3c=..User-Agent: Boto/2.45.0 Python/2.7.12 Linux/4.4.0-31-generic.... #
T 10.144.225.175:8080 -> 10.0.2.15:43350 [A]
...... #
T 10.144.225.175:8080 -> 10.0.2.15:43350 [AP]
HTTP/1.1 200 OK..x-amz-request-id: tx000000000000000001657-0058a124ac-64d335-dev..Content-Length: 517..Date: Mon, 13 Feb 2017 03:14:52 GMT.... ##
T 10.144.225.175:8080 -> 10.0.2.15:43350 [AP] {"entry_point_object_ver":{"tag":"_1JjHz-Vq_JVjwLagDyKLVaP","ver":1},"object_ver":{"tag":"_r-VMOTmvdDtPzT84aLzcN1A","ver":1},"bucket_info":{"bucket":{"name":"my-new-bucket","pool":".rgw.buckets","dat
a_extra_pool":"","index_pool":".rgw.buckets","marker":"dev.6607669.3","bucket_id":"dev.6607669.3"},"creation_time":1486715428,"owner":"johndoe","flags":0,"region":"default","placement_rule":"","has_i
nstance_obj":"true","quota":{"enabled":false,"max_size_kb":-1,"max_objects":-1},"num_shards":0,"bi_shard_hash_type":0}}


Related issues

Copied to rgw - Backport #19171: jewel: rgw: S3 create bucket should not do response in json Resolved
Copied to rgw - Backport #19172: kraken: rgw: S3 create bucket should not do response in json Resolved

History

#1 Updated by Abhishek Lekshmanan 6 months ago

Can you post more detailed logs on how you got the json, boto (both 2 and 3) should work seamlessly with radosgw

#2 Updated by okwap okwap 6 months ago

Abhishek Lekshmanan wrote:

Can you post more detailed logs on how you got the json, boto (both 2 and 3) should work seamlessly with radosgw

boto2 is ok.

reproduce problem code (boto3 (1.4.4)):

import boto3

s3client = boto3.client('s3', endpoint_url='http://X.X.x.X:8080/',
        aws_access_key_id = 'XXXXXXX',
        aws_secret_access_key = 'XXXXXXXX')
bucket_name = 'aaaaa'
s3client.create_bucket(Bucket=bucket_name)

Traceback (most recent call last):

  File "s3_sample.py", line 54, in <module>
    s3client.create_bucket(Bucket=bucket_name)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 530, in _make_api_call
    operation_model, request_dict)
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 168, in _send_request
    request, operation_model, attempts)
  File "/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py", line 233, in _get_response
    response_dict, operation_model.output_shape)
  File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 211, in parse
    parsed = self._do_parse(response, shape)
  File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 613, in _do_parse
    self._parse_payload(response, shape, member_shapes, final_parsed)
  File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 646, in _parse_payload
    original_parsed = self._initial_body_parse(response['body'])
  File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 721, in _initial_body_parse
    return self._parse_xml_string_to_dom(xml_string)
  File "/usr/local/lib/python2.7/dist-packages/botocore/parsers.py", line 389, in _parse_xml_string_to_dom
    "invalid XML received:\n%s" % (e, xml_string))
botocore.parsers.ResponseParserError: Unable to parse response (not well-formed (invalid token): line 1, column 0), invalid XML received:
{"entry_point_object_ver":{"tag":"_0we_De_tZIlMsTJr2IFlbb-","ver":1},"object_ver":{"tag":"_BTTaYk--57eQwS3zXU6E1an","ver":1},"bucket_info":{"bucket":{"name":"aaaaa","pool":".rgw.buckets","data_extra_pool":"","index_pool":".rgw.buckets","marker":"dev.6607669.11","bucket_id":"dev.6607669.11"},"creation_time":1486957510,"owner":"johndoe","flags":0,"region":"default","placement_rule":"","has_instance_obj":"true","quota":{"enabled":false,"max_size_kb":-1,"max_objects":-1},"num_shards":0,"bi_shard_hash_type":0}}

#3 Updated by Abhishek Lekshmanan 6 months ago

okwap okwap wrote:

Abhishek Lekshmanan wrote:

Can you post more detailed logs on how you got the json, boto (both 2 and 3) should work seamlessly with radosgw

boto2 is ok.

reproduce problem code (boto3 (1.4.4)):
[...]

Traceback (most recent call last):
[...]

Hm, this is because you requested a create_bucket as a system user, which is actually meant primarily for zone sync operations etc.

#4 Updated by Abhishek Lekshmanan 6 months ago

Maybe we should add a warning in the docs mentioning about this

#5 Updated by Abhishek Lekshmanan 6 months ago

I've created https://github.com/ceph/ceph/pull/13461 to make a note in docs.

#6 Updated by Abhishek Lekshmanan 6 months ago

  • Status changed from New to In Progress
  • Assignee set to Abhishek Lekshmanan

#7 Updated by okwap okwap 6 months ago

Cool.

#8 Updated by Abhishek Lekshmanan 6 months ago

  • Tracker changed from Bug to Documentation
  • Status changed from In Progress to Pending Backport
  • Backport set to jewel,kraken

#9 Updated by Nathan Cutler 6 months ago

  • Copied to Backport #19171: jewel: rgw: S3 create bucket should not do response in json added

#10 Updated by Nathan Cutler 6 months ago

  • Copied to Backport #19172: kraken: rgw: S3 create bucket should not do response in json added

#11 Updated by Nathan Cutler 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF