Bug #23188
closedCannot list file in a bucket or delete it : illegal character code U+0001
0%
Description
Hello,
I created for the first time a new bucket since my upgrade to luminous then upload files using rclone sync. At the end of the sync, it was impossible to list files, I can list files separatly1 but not on the bucket itself2. I tried to delete the bucket but even I think I have deleted all the files, it's still remain object that disallow me to remove it.
(1)
root@/prod/toCopy/spinn3r.json ~ for i in * ; do s3cmd -c ~/.s3cfg-admin ls s3://spinn3r.json/$i ; done 2018-02-26 12:57 21004796524 s3://spinn3r.json/part-00000 2018-02-26 12:56 19095193372 s3://spinn3r.json/part-00001 2018-02-26 12:57 22112021746 s3://spinn3r.json/part-00002 2018-02-26 12:56 16981566299 s3://spinn3r.json/part-00003 2018-02-26 13:05 44200404714 s3://spinn3r.json/part-00004 2018-02-26 12:54 12043548278 s3://spinn3r.json/part-00005 2018-02-26 12:57 20445411480 s3://spinn3r.json/part-00006 2018-02-26 12:57 20314776396 s3://spinn3r.json/part-00007 2018-02-26 13:00 28017331807 s3://spinn3r.json/part-00008 2018-02-26 13:03 38032014310 s3://spinn3r.json/part-00009 2018-02-26 13:03 38346534961 s3://spinn3r.json/part-00010 2018-02-26 13:00 27998655031 s3://spinn3r.json/part-00011 2018-02-26 13:02 21128807819 s3://spinn3r.json/part-00012 2018-02-26 13:12 43336123754 s3://spinn3r.json/part-00013 2018-02-26 13:07 29826924759 s3://spinn3r.json/part-00014 2018-02-26 13:05 22560405095 s3://spinn3r.json/part-00015 2018-02-26 13:08 30933437627 s3://spinn3r.json/part-00016 2018-02-26 13:15 49775102432 s3://spinn3r.json/part-00017 2018-02-26 13:08 29669512621 s3://spinn3r.json/part-00018 2018-02-26 13:11 30573323487 s3://spinn3r.json/part-00019 2018-02-26 13:08 21204630126 s3://spinn3r.json/part-00020 2018-02-26 13:13 30496402902 s3://spinn3r.json/part-00021 2018-02-26 13:13 24106425792 s3://spinn3r.json/part-00022 2018-02-26 13:14 27276009858 s3://spinn3r.json/part-00023 2018-02-26 13:13 19468991890 s3://spinn3r.json/part-00024 2018-02-26 13:14 22408058693 s3://spinn3r.json/part-00025 2018-02-26 13:16 20889341088 s3://spinn3r.json/part-00026 2018-02-26 13:18 26133522525 s3://spinn3r.json/part-00027 2018-02-26 13:22 35086066316 s3://spinn3r.json/part-00028 2018-02-26 13:15 18150592007 s3://spinn3r.json/part-00029 2018-02-26 13:24 33472457032 s3://spinn3r.json/part-00030 2018-02-26 13:20 19424233519 s3://spinn3r.json/part-00031 2018-02-26 13:21 22006474055 s3://spinn3r.json/part-00032 2018-02-26 13:26 34081216684 s3://spinn3r.json/part-00033 2018-02-26 13:23 24600403175 s3://spinn3r.json/part-00034 2018-02-26 13:28 37325347582 s3://spinn3r.json/part-00035 2018-02-26 13:19 13745022092 s3://spinn3r.json/part-00036 2018-02-26 13:25 25020342768 s3://spinn3r.json/part-00037 2018-02-26 13:21 14800136497 s3://spinn3r.json/part-00038 2018-02-26 13:22 16934267005 s3://spinn3r.json/part-00039 2018-02-26 13:26 20513126537 s3://spinn3r.json/part-00040 2018-02-26 13:30 26864219697 s3://spinn3r.json/part-00041 2018-02-26 13:26 14547526756 s3://spinn3r.json/part-00042 2018-02-26 13:30 22261621086 s3://spinn3r.json/part-00043 2018-02-26 13:30 21932891577 s3://spinn3r.json/part-00044 2018-02-26 12:48 0 s3://spinn3r.json/_SUCCESS
(2)
root@~ $ rclone -vvvv ls prodadmin:spinn3r.json 2018/03/01 13:33:57 DEBUG : rclone: Version "v1.37" starting with parameters ["rclone" "-vvvv" "ls" "prodadmin:spinn3r.json"] 2018/03/01 13:33:57 DEBUG : prodguest: Using v2 auth 2018/03/01 13:33:57 INFO : S3 bucket spinn3r.json: Modify window is 1ns 2018/03/01 13:33:57 ERROR : : error listing: SerializationError: failed to decode REST XML response caused by: XML syntax error on line 1: illegal character code U+0001 2018/03/01 13:33:57 Failed to ls: SerializationError: failed to decode REST XML response caused by: XML syntax error on line 1: illegal character code U+0001
Some test I did and result of commands :
root@~ $ s3cmd -c .s3cfg-admin ls s3://spinn3r.json ERROR: Error parsing xml: reference to invalid character number: line 1, column 214 <file in attachment> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions If the error persists, please report the following lines (removing any private info as necessary) to: s3tools-bugs@lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Invoked as: /usr/local/bin/s3cmd -c .s3cfg-admin ls s3://spinn3r.json Problem: ParseError: reference to invalid character number: line 1, column 214 S3cmd: 2.0.1 python: 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] environment LANG=en_US.UTF-8 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 3073, in <module> File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2989, in main File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 140, in cmd_ls File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 176, in subcmd_bucket_list File "build/bdist.linux-x86_64/egg/S3/S3.py", line 308, in bucket_list for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit): File "build/bdist.linux-x86_64/egg/S3/S3.py", line 343, in bucket_list_streaming current_list = _get_contents(response["data"]) File "build/bdist.linux-x86_64/egg/S3/S3.py", line 326, in _get_contents return getListFromXml(data, "Contents") File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 117, in getListFromXml tree = getTreeFromXml(xml) File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 105, in getTreeFromXml tree = ET.fromstring(xml) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML parser.feed(text) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed self._raiseerror(v) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror raise err ParseError: reference to invalid character number: line 1, column 214 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions If the error persists, please report the above lines (removing any private info as necessary) to: s3tools-bugs@lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
root@~ $ s3cmd -c .s3cfg-admin rm -r --force s3://spinn3r.json ERROR: Error parsing xml: reference to invalid character number: line 1, column 190 Invoked as: /usr/local/bin/s3cmd -c .s3cfg-admin rm -r --force s3://spinn3r.json Problem: ParseError: reference to invalid character number: line 1, column 190 S3cmd: 2.0.1 python: 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609] environment LANG=en_US.UTF-8 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 3073, in <module> File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2989, in main File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 640, in cmd_object_del File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 662, in subcmd_batch_del_iterative File "build/bdist.linux-x86_64/egg/S3/S3.py", line 343, in bucket_list_streaming current_list = _get_contents(response["data"]) File "build/bdist.linux-x86_64/egg/S3/S3.py", line 326, in _get_contents return getListFromXml(data, "Contents") File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 117, in getListFromXml tree = getTreeFromXml(xml) File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 105, in getTreeFromXml tree = ET.fromstring(xml) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML parser.feed(text) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed self._raiseerror(v) File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror raise err ParseError: reference to invalid character number: line 1, column 190 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please try reproducing the error using the latest s3cmd code from the git master branch found at: https://github.com/s3tools/s3cmd and have a look at the known issues list: https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions If the error persists, please report the above lines (removing any private info as necessary) to: s3tools-bugs@lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
root@~ $ rclone -v rmdir prodsadmin:spinn3r.json 2018/03/01 13:56:23 INFO : S3 bucket spinn3r.json: Modify window is 1ns 2018/03/01 13:56:23 ERROR : Attempt 1/3 failed with 1 errors and: BucketNotEmpty: status code: 409, request id: tx000000000000000000278-005a97f877-17647d8-default, host id: 2018/03/01 13:56:23 ERROR : Attempt 2/3 failed with 1 errors and: BucketNotEmpty: status code: 409, request id: tx000000000000000000279-005a97f877-17647d8-default, host id: 2018/03/01 13:56:23 ERROR : Attempt 3/3 failed with 1 errors and: BucketNotEmpty: status code: 409, request id: tx00000000000000000027a-005a97f877-17647d8-default, host id: 2018/03/01 13:56:23 Failed to rmdir: BucketNotEmpty: status code: 409, request id: tx00000000000000000027a-005a97f877-17647d8-default, host id:
cephprod@icadmin004:~$ radosgw-admin bucket rm --bucket=spinn3r.json 2018-03-01 13:41:55.493278 7fb5fdf74cc0 -1 ERROR: could not remove non-empty bucket spinn3r.json 2018-03-01 13:41:55.493359 7fb5fdf74cc0 -1 ERROR: unable to remove bucket(39) Directory not empty
cephprod@icadmin004:~$ radosgw-admin bucket stats --bucket=spinn3r.json { "bucket": "spinn3r.json", "zonegroup": "4d982760-7853-4174-8c05-cec2ef148cf0", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "c9724aff-5fa0-4dd9-b494-57bdb48fab4e.24412860.1", "marker": "c9724aff-5fa0-4dd9-b494-57bdb48fab4e.24412860.1", "index_type": "Normal", "owner": "admin", "ver": "0#443328", "master_ver": "0#0", "mtime": "2018-02-26 13:48:20.340155", "max_marker": "0#", "usage": { "rgw.main": { "size": 1226840320, "size_actual": 1227358208, "size_utilized": 1226840320, "size_kb": 1198087, "size_kb_actual": 1198592, "size_kb_utilized": 1198087, "num_objects": 362 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 4 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }
cephprod@icadmin004:~$ radosgw-admin bucket check --bucket=spinn3r.json --check-objects --fix [] { "object": "0x01-\u0001-", "object": "0x02-\u0002-", "object": "0x03-\u0003-", "object": "0x04-\u0004-", "object": "0x05-\u0005-", "object": "0x06-\u0006-", "object": "0x07-\u0007-", "object": "0x08-\u0008-", "object": "0x09-\t-", "object": "0x0A-\n-", "object": "0x0B-\u000b-", "object": "0x0C-\u000c-", "object": "0x0D-\u000d-", "object": "0x0E-\u000e-", "object": "0x0F-\u000f-", "object": "0x10-\u0010-", "object": "0x11-\u0011-", "object": "0x12-\u0012-", "object": "0x13-\u0013-", "object": "0x14-\u0014-", "object": "0x15-\u0015-", "object": "0x16-\u0016-", "object": "0x17-\u0017-", "object": "0x18-\u0018-", "object": "0x19-\u0019-", "object": "0x1A-\u001a-", "object": "0x1B-\u001b-", "object": "0x1C-\u001c-", "object": "0x1D-\u001d-", "object": "0x1E-\u001e-", "object": "0x1F-\u001f-", "object": "0x20- -", "object": "0x21-!-", "object": "0x22-\"-", "object": "0x23-#-", "object": "0x24-$-", "object": "0x25-%-", "object": "0x26-&-", "object": "0x27-'-", "object": "0x28-(-", "object": "0x29-)-", "object": "0x2A-*-", "object": "0x2B-+-", "object": "0x2C-,-", "object": "0x2D---", "object": "0x2E-.-", "object": "0x30-0-", "object": "0x31-1-", "object": "0x32-2-", "object": "0x33-3-", "object": "0x34-4-", "object": "0x35-5-", "object": "0x36-6-", "object": "0x37-7-", "object": "0x38-8-", "object": "0x39-9-", "object": "0x3A-:-", "object": "0x3B-;-", "object": "0x3C-<-", "object": "0x3D-=-", "object": "0x3E->-", "object": "0x3F-?-", "object": "0x40-@-", "object": "0x41-A-", "object": "0x42-B-", "object": "0x43-C-", "object": "0x44-D-", "object": "0x45-E-", "object": "0x46-F-", "object": "0x47-G-", "object": "0x48-H-", "object": "0x49-I-", "object": "0x4A-J-", "object": "0x4B-K-", "object": "0x4C-L-", "object": "0x4D-M-", "object": "0x4E-N-", "object": "0x4F-O-", "object": "0x50-P-", "object": "0x51-Q-", "object": "0x52-R-", "object": "0x53-S-", "object": "0x54-T-", "object": "0x55-U-", "object": "0x56-V-", "object": "0x57-W-", "object": "0x58-X-", "object": "0x59-Y-", "object": "0x5A-Z-", "object": "0x5B-[-", "object": "0x5C-\\-", "object": "0x5D-]-", "object": "0x5E-^-", "object": "0x5F-_-", "object": "0x60-`-", "object": "0x61-a-", "object": "0x62-b-", "object": "0x63-c-", "object": "0x64-d-", "object": "0x65-e-", "object": "0x66-f-", "object": "0x67-g-", "object": "0x68-h-", "object": "0x69-i-", "object": "0x6A-j-", "object": "0x6B-k-", "object": "0x6C-l-", "object": "0x6D-m-", "object": "0x6E-n-", "object": "0x6F-o-", "object": "0x70-p-", "object": "0x71-q-", "object": "0x72-r-", "object": "0x73-s-", "object": "0x74-t-", "object": "0x75-u-", "object": "0x76-v-", "object": "0x77-w-", "object": "0x78-x-", "object": "0x79-y-", "object": "0x7A-z-", "object": "0x7B-{-", "object": "0x7C-|-", "object": "0x7D-}-", "object": "0x7E-~-", "object": "0x7F-\u007f-", "object": "Héroique", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.1", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.10", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.11", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.12", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.13", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.14", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.15", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.16", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.17", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.18", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.19", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.2", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.20", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.21", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.22", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.23", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.24", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.25", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.26", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.27", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.28", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.29", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.3", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.30", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.31", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.32", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.33", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.34", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.35", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.36", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.37", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.38", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.39", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.4", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.40", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.41", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.42", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.43", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.44", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.45", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.46", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.47", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.48", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.5", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.6", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.7", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.8", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.9", "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.meta", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.1", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.10", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.11", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.12", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.13", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.14", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.15", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.16", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.17", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.18", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.19", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.2", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.20", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.21", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.22", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.23", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.24", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.25", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.26", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.27", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.28", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.29", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.3", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.30", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.31", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.32", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.33", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.34", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.35", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.36", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.37", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.38", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.39", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.4", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.40", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.41", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.42", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.43", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.44", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.45", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.46", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.47", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.48", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.49", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.5", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.50", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.51", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.52", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.53", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.54", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.55", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.56", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.57", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.58", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.59", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.6", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.60", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.61", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.62", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.63", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.64", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.65", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.66", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.67", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.68", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.7", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.8", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.9", "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.meta", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.1", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.10", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.11", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.12", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.13", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.14", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.15", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.16", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.17", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.18", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.19", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.2", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.20", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.21", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.22", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.23", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.24", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.25", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.26", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.27", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.28", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.29", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.3", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.30", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.31", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.32", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.33", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.34", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.35", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.36", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.37", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.38", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.39", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.4", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.40", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.41", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.42", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.43", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.44", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.45", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.46", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.47", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.48", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.49", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.5", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.50", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.51", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.52", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.53", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.54", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.55", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.56", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.57", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.58", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.59", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.6", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.60", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.61", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.62", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.63", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.64", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.65", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.66", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.67", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.68", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.69", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.7", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.8", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.9", "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.meta", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.1", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.10", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.11", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.12", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.13", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.14", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.15", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.16", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.17", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.18", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.19", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.2", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.20", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.21", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.22", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.23", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.24", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.25", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.26", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.27", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.28", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.29", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.3", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.30", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.31", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.32", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.33", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.34", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.35", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.36", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.37", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.38", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.39", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.4", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.40", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.41", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.42", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.43", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.44", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.45", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.46", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.47", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.48", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.49", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.5", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.6", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.7", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.8", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.9", "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.meta", "object": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" } { "existing_header": { "usage": { "rgw.main": { "size": 1226840320, "size_actual": 1227358208, "size_utilized": 1226840320, "size_kb": 1198087, "size_kb_actual": 1198592, "size_kb_utilized": 1198087, "num_objects": 362 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 4 } } }, "calculated_header": { "usage": { "rgw.main": { "size": 1226840320, "size_actual": 1227358208, "size_utilized": 1226840320, "size_kb": 1198087, "size_kb_actual": 1198592, "size_kb_utilized": 1198087, "num_objects": 362 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 4 } } } }
Files
Updated by Yoann Moulin about 6 years ago
- File cephprod-rgw-log.tgz cephprod-rgw-log.tgz added
some additional logs fronm radosgw
Updated by Yoann Moulin about 6 years ago
I found plenty of "execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json" in radosgw logs
2018-03-01 11:03:48.436376 7feac1218700 1 civetweb: 0x55cad4f93000: 127.0.0.1 - - [01/Mar/2018:11:03:48 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:14.757214 7feab7a05700 1 civetweb: 0x55cad4ffc000: 127.0.0.1 - - [01/Mar/2018:11:08:14 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:14.775363 7fead5a41700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:08:18.647447 7feab8206700 1 civetweb: 0x55cad4ff7000: 127.0.0.1 - - [01/Mar/2018:11:08:18 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:18.663724 7feaca22a700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:08:26.920768 7feaa81e6700 1 civetweb: 0x55cad509b000: 127.0.0.1 - - [01/Mar/2018:11:08:26 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:26.932970 7feac1a19700 1 civetweb: 0x55cad4f8e000: 127.0.0.1 - - [01/Mar/2018:11:08:26 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:26.950050 7feaae9f3700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.141228 7feac7a25700 1 civetweb: 0x55cad4f4e000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.157647 7feaa99e9700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.387137 7feabfa15700 1 civetweb: 0x55cad4fac000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.401582 7fead1238700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.625722 7feabb20c700 1 civetweb: 0x55cad4fd9000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.636035 7feaa69e3700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.848721 7fead2a3b700 1 civetweb: 0x55cad4ed4000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.863566 7feaae1f2700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.087841 7feacca2f700 1 civetweb: 0x55cad4f17000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.104343 7feac6222700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.337346 7feac8226700 1 civetweb: 0x55cad4f49000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.352437 7fead0a37700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.580932 7feaa89e7700 1 civetweb: 0x55cad5096000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.595777 7feaab9ed700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
Updated by Pritha Srivastava about 6 years ago
Can you please increase the logging level to 20 and then attach logs.
Updated by Yoann Moulin about 6 years ago
ceph-post-file: 26c152cf-1fa8-4ef8-9bb8-1621bfcd43bc
Updated by Pritha Srivastava about 6 years ago
Did you attach any file? Sorry I didnt get it.
Updated by Pritha Srivastava about 6 years ago
Hi Yoann, Can you please attach the log files here. Not sure how to get the files that you uploaded using ceph-post-file. Or if you know how to get them, please let me know.
Updated by Yoann Moulin about 6 years ago
here the log file
Updated by Pritha Srivastava about 6 years ago
Hi Yoann, I checked all the log files including the latest one, but I dont see any errors there. I also tried to reproduce the issue by uploading object with utf-8 chars in its name, but i wasnt able to reproduce the issue. If you can give me the exact steps to repro the issue then it will help.
Updated by Yoann Moulin about 6 years ago
That's really strange, I have a dataset on a rbd device on the same cluster, I just did.
rclone mkdir account:dataset
rclone copy dataset/ account:datatset/
Everything went well, but at the end of the copy, I got this error and weren't able to list the content of the dataset due to this error. I was able to list files separately. After those errors, I decided to remove the bucket without success, I have been able to delete files in the bucket one by one but it still impossible to remove this bucket.
I have no idea what happened, I'll install a test cluster and try to upload this dataset again and see if this issue happens again.
Updated by Pritha Srivastava about 6 years ago
The last log file that you had attached with increased log levels, had two operations:
1. Delete Bucket, which failed with a Directory Not Empty error, because the objects were not deleted from it
2018-03-06 17:22:27.488380 7f2bd9b33700 2 req 1:0.011615:s3:DELETE /spinn3r.json:delete_bucket:verifying op params
2018-03-06 17:22:27.488384 7f2bd9b33700 2 req 1:0.011617:s3:DELETE /spinn3r.json:delete_bucket:pre-executing
2018-03-06 17:22:27.488387 7f2bd9b33700 2 req 1:0.011622:s3:DELETE /spinn3r.json:delete_bucket:executing
...
2018-03-06 17:22:27.510499 7f2bd9b33700 2 req 1:0.033733:s3:DELETE /spinn3r.json:delete_bucket:completing
2018-03-06 17:22:27.510625 7f2bd9b33700 2 req 1:0.033859:s3:DELETE /spinn3r.json:delete_bucket:op status=-39
2018-03-06 17:22:27.510629 7f2bd9b33700 2 req 1:0.033864:s3:DELETE /spinn3r.json:delete_bucket:http status=409
2018-03-06 17:22:27.510631 7f2bd9b33700 1 ====== req done req=0x7f2bd9b2d1f0 op status=-39 http_status=409 ======
2. Get Bucket, which succeeded
2018-03-06 17:22:58.499629 7f2bd2b25700 2 req 5:0.000608:s3:GET /:list_bucket:verifying op params
2018-03-06 17:22:58.499632 7f2bd2b25700 2 req 5:0.000610:s3:GET /:list_bucket:pre-executing
2018-03-06 17:22:58.499634 7f2bd2b25700 2 req 5:0.000613:s3:GET /:list_bucket:executing
...
2018-03-06 17:22:58.510320 7f2bd2b25700 2 req 5:0.011299:s3:GET /:list_bucket:http status=200
2018-03-06 17:22:58.510403 7f2bd2b25700 1 ====== req done req=0x7f2bd2b1f1f0 op status=0 http_status=200 ======
I didnt see any error in the bucket list operation. The Delete bucket operation failed because the bucket still had objects in it. But apart from these two operations, none of the other requests (like deleting every object individually), was logged. So it is difficult to conclude why the bucket wasn't empty.
Updated by Yoann Moulin about 6 years ago
I just uploaded files of the bucket on a new fresh ceph cluster and everything is fine, I can list bucket files without any issue.
the next step for me is, How I can delete this bucket to be able to re upload it with the same name.
Yoann
Updated by Yoann Moulin about 6 years ago
I didn't think to use --purge-objects before :
radosgw-admin bucket rm --purge-objects --bucket=<bucket>
With this option, I have been able to rm the bucket, recreate it and upload files again, I didn't get any error and I'm able to list files. I have no idea what happened but now it works for me, we can close this issue.
Thanks for your help
Yoann
Updated by Yehuda Sadeh about 6 years ago
I had a similar issue in a dev branch. The problem was that for some reason one of the properties in bucket list operation got extra junk. The culprit was a bit tricky to identify, but it ended up being a racing bucket list and dir_suggest, and the dir_suggest was feeding in bad data. The reason for it was that in my branch I removed the extra null character of the etag field, and dir_suggest accessed that xattr using bufferlist.c_str(). It could be that we have the same issue with some other field (e.g., content_type).