Project

General

Profile

Bug #23188

Cannot list file in a bucket or delete it : illegal character code U+0001

Added by Yoann Moulin about 6 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
High
Target version:
% Done:

0%

Source:
Community (user)
Tags:
radosgw
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello,

I created for the first time a new bucket since my upgrade to luminous then upload files using rclone sync. At the end of the sync, it was impossible to list files, I can list files separatly1 but not on the bucket itself2. I tried to delete the bucket but even I think I have deleted all the files, it's still remain object that disallow me to remove it.

(1)

root@/prod/toCopy/spinn3r.json ~ for i in * ; do s3cmd -c ~/.s3cfg-admin ls s3://spinn3r.json/$i ; done
2018-02-26 12:57 21004796524   s3://spinn3r.json/part-00000
2018-02-26 12:56 19095193372   s3://spinn3r.json/part-00001
2018-02-26 12:57 22112021746   s3://spinn3r.json/part-00002
2018-02-26 12:56 16981566299   s3://spinn3r.json/part-00003
2018-02-26 13:05 44200404714   s3://spinn3r.json/part-00004
2018-02-26 12:54 12043548278   s3://spinn3r.json/part-00005
2018-02-26 12:57 20445411480   s3://spinn3r.json/part-00006
2018-02-26 12:57 20314776396   s3://spinn3r.json/part-00007
2018-02-26 13:00 28017331807   s3://spinn3r.json/part-00008
2018-02-26 13:03 38032014310   s3://spinn3r.json/part-00009
2018-02-26 13:03 38346534961   s3://spinn3r.json/part-00010
2018-02-26 13:00 27998655031   s3://spinn3r.json/part-00011
2018-02-26 13:02 21128807819   s3://spinn3r.json/part-00012
2018-02-26 13:12 43336123754   s3://spinn3r.json/part-00013
2018-02-26 13:07 29826924759   s3://spinn3r.json/part-00014
2018-02-26 13:05 22560405095   s3://spinn3r.json/part-00015
2018-02-26 13:08 30933437627   s3://spinn3r.json/part-00016
2018-02-26 13:15 49775102432   s3://spinn3r.json/part-00017
2018-02-26 13:08 29669512621   s3://spinn3r.json/part-00018
2018-02-26 13:11 30573323487   s3://spinn3r.json/part-00019
2018-02-26 13:08 21204630126   s3://spinn3r.json/part-00020
2018-02-26 13:13 30496402902   s3://spinn3r.json/part-00021
2018-02-26 13:13 24106425792   s3://spinn3r.json/part-00022
2018-02-26 13:14 27276009858   s3://spinn3r.json/part-00023
2018-02-26 13:13 19468991890   s3://spinn3r.json/part-00024
2018-02-26 13:14 22408058693   s3://spinn3r.json/part-00025
2018-02-26 13:16 20889341088   s3://spinn3r.json/part-00026
2018-02-26 13:18 26133522525   s3://spinn3r.json/part-00027
2018-02-26 13:22 35086066316   s3://spinn3r.json/part-00028
2018-02-26 13:15 18150592007   s3://spinn3r.json/part-00029
2018-02-26 13:24 33472457032   s3://spinn3r.json/part-00030
2018-02-26 13:20 19424233519   s3://spinn3r.json/part-00031
2018-02-26 13:21 22006474055   s3://spinn3r.json/part-00032
2018-02-26 13:26 34081216684   s3://spinn3r.json/part-00033
2018-02-26 13:23 24600403175   s3://spinn3r.json/part-00034
2018-02-26 13:28 37325347582   s3://spinn3r.json/part-00035
2018-02-26 13:19 13745022092   s3://spinn3r.json/part-00036
2018-02-26 13:25 25020342768   s3://spinn3r.json/part-00037
2018-02-26 13:21 14800136497   s3://spinn3r.json/part-00038
2018-02-26 13:22 16934267005   s3://spinn3r.json/part-00039
2018-02-26 13:26 20513126537   s3://spinn3r.json/part-00040
2018-02-26 13:30 26864219697   s3://spinn3r.json/part-00041
2018-02-26 13:26 14547526756   s3://spinn3r.json/part-00042
2018-02-26 13:30 22261621086   s3://spinn3r.json/part-00043
2018-02-26 13:30 21932891577   s3://spinn3r.json/part-00044
2018-02-26 12:48         0   s3://spinn3r.json/_SUCCESS

(2)

root@~ $ rclone -vvvv ls prodadmin:spinn3r.json
2018/03/01 13:33:57 DEBUG : rclone: Version "v1.37" starting with parameters ["rclone" "-vvvv" "ls" "prodadmin:spinn3r.json"]
2018/03/01 13:33:57 DEBUG : prodguest: Using v2 auth
2018/03/01 13:33:57 INFO  : S3 bucket spinn3r.json: Modify window is 1ns
2018/03/01 13:33:57 ERROR : : error listing: SerializationError: failed to decode REST XML response
caused by: XML syntax error on line 1: illegal character code U+0001
2018/03/01 13:33:57 Failed to ls: SerializationError: failed to decode REST XML response
caused by: XML syntax error on line 1: illegal character code U+0001

Some test I did and result of commands :

root@~ $ s3cmd -c .s3cfg-admin ls s3://spinn3r.json
ERROR: Error parsing xml: reference to invalid character number: line 1, column 214
<file in attachment>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd -c .s3cfg-admin ls s3://spinn3r.json
Problem: ParseError: reference to invalid character number: line 1, column 214
S3cmd:   2.0.1
python:   2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 3073, in <module>
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2989, in main
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 140, in cmd_ls
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 176, in subcmd_bucket_list
  File "build/bdist.linux-x86_64/egg/S3/S3.py", line 308, in bucket_list
    for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
  File "build/bdist.linux-x86_64/egg/S3/S3.py", line 343, in bucket_list_streaming
    current_list = _get_contents(response["data"])
  File "build/bdist.linux-x86_64/egg/S3/S3.py", line 326, in _get_contents
    return getListFromXml(data, "Contents")
  File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 117, in getListFromXml
    tree = getTreeFromXml(xml)
  File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 105, in getTreeFromXml
    tree = ET.fromstring(xml)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: reference to invalid character number: line 1, column 214

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
root@~ $ s3cmd -c .s3cfg-admin rm -r --force s3://spinn3r.json
ERROR: Error parsing xml: reference to invalid character number: line 1, column 190

Invoked as: /usr/local/bin/s3cmd -c .s3cfg-admin rm -r --force s3://spinn3r.json
Problem: ParseError: reference to invalid character number: line 1, column 190
S3cmd:   2.0.1
python:   2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609]
environment LANG=en_US.UTF-8

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 3073, in <module>
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 2989, in main
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 640, in cmd_object_del
  File "/usr/local/lib/python2.7/dist-packages/s3cmd-2.0.1-py2.7.egg/EGG-INFO/scripts/s3cmd", line 662, in subcmd_batch_del_iterative
  File "build/bdist.linux-x86_64/egg/S3/S3.py", line 343, in bucket_list_streaming
    current_list = _get_contents(response["data"])
  File "build/bdist.linux-x86_64/egg/S3/S3.py", line 326, in _get_contents
    return getListFromXml(data, "Contents")
  File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 117, in getListFromXml
    tree = getTreeFromXml(xml)
  File "build/bdist.linux-x86_64/egg/S3/Utils.py", line 105, in getTreeFromXml
    tree = ET.fromstring(xml)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: reference to invalid character number: line 1, column 190

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
root@~ $ rclone -v rmdir prodsadmin:spinn3r.json
2018/03/01 13:56:23 INFO  : S3 bucket spinn3r.json: Modify window is 1ns
2018/03/01 13:56:23 ERROR : Attempt 1/3 failed with 1 errors and: BucketNotEmpty: 
    status code: 409, request id: tx000000000000000000278-005a97f877-17647d8-default, host id: 
2018/03/01 13:56:23 ERROR : Attempt 2/3 failed with 1 errors and: BucketNotEmpty: 
    status code: 409, request id: tx000000000000000000279-005a97f877-17647d8-default, host id: 
2018/03/01 13:56:23 ERROR : Attempt 3/3 failed with 1 errors and: BucketNotEmpty: 
    status code: 409, request id: tx00000000000000000027a-005a97f877-17647d8-default, host id: 
2018/03/01 13:56:23 Failed to rmdir: BucketNotEmpty: 
    status code: 409, request id: tx00000000000000000027a-005a97f877-17647d8-default, host id: 

cephprod@icadmin004:~$ radosgw-admin bucket rm --bucket=spinn3r.json
2018-03-01 13:41:55.493278 7fb5fdf74cc0 -1 ERROR: could not remove non-empty bucket spinn3r.json
2018-03-01 13:41:55.493359 7fb5fdf74cc0 -1 ERROR: unable to remove bucket(39) Directory not empty
cephprod@icadmin004:~$ radosgw-admin bucket stats --bucket=spinn3r.json
{
    "bucket": "spinn3r.json",
    "zonegroup": "4d982760-7853-4174-8c05-cec2ef148cf0",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": "" 
    },
    "id": "c9724aff-5fa0-4dd9-b494-57bdb48fab4e.24412860.1",
    "marker": "c9724aff-5fa0-4dd9-b494-57bdb48fab4e.24412860.1",
    "index_type": "Normal",
    "owner": "admin",
    "ver": "0#443328",
    "master_ver": "0#0",
    "mtime": "2018-02-26 13:48:20.340155",
    "max_marker": "0#",
    "usage": {
        "rgw.main": {
            "size": 1226840320,
            "size_actual": 1227358208,
            "size_utilized": 1226840320,
            "size_kb": 1198087,
            "size_kb_actual": 1198592,
            "size_kb_utilized": 1198087,
            "num_objects": 362
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 4
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
cephprod@icadmin004:~$ radosgw-admin bucket check --bucket=spinn3r.json --check-objects --fix
[]
{
    "object": "0x01-\u0001-",
    "object": "0x02-\u0002-",
    "object": "0x03-\u0003-",
    "object": "0x04-\u0004-",
    "object": "0x05-\u0005-",
    "object": "0x06-\u0006-",
    "object": "0x07-\u0007-",
    "object": "0x08-\u0008-",
    "object": "0x09-\t-",
    "object": "0x0A-\n-",
    "object": "0x0B-\u000b-",
    "object": "0x0C-\u000c-",
    "object": "0x0D-\u000d-",
    "object": "0x0E-\u000e-",
    "object": "0x0F-\u000f-",
    "object": "0x10-\u0010-",
    "object": "0x11-\u0011-",
    "object": "0x12-\u0012-",
    "object": "0x13-\u0013-",
    "object": "0x14-\u0014-",
    "object": "0x15-\u0015-",
    "object": "0x16-\u0016-",
    "object": "0x17-\u0017-",
    "object": "0x18-\u0018-",
    "object": "0x19-\u0019-",
    "object": "0x1A-\u001a-",
    "object": "0x1B-\u001b-",
    "object": "0x1C-\u001c-",
    "object": "0x1D-\u001d-",
    "object": "0x1E-\u001e-",
    "object": "0x1F-\u001f-",
    "object": "0x20- -",
    "object": "0x21-!-",
    "object": "0x22-\"-",
    "object": "0x23-#-",
    "object": "0x24-$-",
    "object": "0x25-%-",
    "object": "0x26-&-",
    "object": "0x27-'-",
    "object": "0x28-(-",
    "object": "0x29-)-",
    "object": "0x2A-*-",
    "object": "0x2B-+-",
    "object": "0x2C-,-",
    "object": "0x2D---",
    "object": "0x2E-.-",
    "object": "0x30-0-",
    "object": "0x31-1-",
    "object": "0x32-2-",
    "object": "0x33-3-",
    "object": "0x34-4-",
    "object": "0x35-5-",
    "object": "0x36-6-",
    "object": "0x37-7-",
    "object": "0x38-8-",
    "object": "0x39-9-",
    "object": "0x3A-:-",
    "object": "0x3B-;-",
    "object": "0x3C-<-",
    "object": "0x3D-=-",
    "object": "0x3E->-",
    "object": "0x3F-?-",
    "object": "0x40-@-",
    "object": "0x41-A-",
    "object": "0x42-B-",
    "object": "0x43-C-",
    "object": "0x44-D-",
    "object": "0x45-E-",
    "object": "0x46-F-",
    "object": "0x47-G-",
    "object": "0x48-H-",
    "object": "0x49-I-",
    "object": "0x4A-J-",
    "object": "0x4B-K-",
    "object": "0x4C-L-",
    "object": "0x4D-M-",
    "object": "0x4E-N-",
    "object": "0x4F-O-",
    "object": "0x50-P-",
    "object": "0x51-Q-",
    "object": "0x52-R-",
    "object": "0x53-S-",
    "object": "0x54-T-",
    "object": "0x55-U-",
    "object": "0x56-V-",
    "object": "0x57-W-",
    "object": "0x58-X-",
    "object": "0x59-Y-",
    "object": "0x5A-Z-",
    "object": "0x5B-[-",
    "object": "0x5C-\\-",
    "object": "0x5D-]-",
    "object": "0x5E-^-",
    "object": "0x5F-_-",
    "object": "0x60-`-",
    "object": "0x61-a-",
    "object": "0x62-b-",
    "object": "0x63-c-",
    "object": "0x64-d-",
    "object": "0x65-e-",
    "object": "0x66-f-",
    "object": "0x67-g-",
    "object": "0x68-h-",
    "object": "0x69-i-",
    "object": "0x6A-j-",
    "object": "0x6B-k-",
    "object": "0x6C-l-",
    "object": "0x6D-m-",
    "object": "0x6E-n-",
    "object": "0x6F-o-",
    "object": "0x70-p-",
    "object": "0x71-q-",
    "object": "0x72-r-",
    "object": "0x73-s-",
    "object": "0x74-t-",
    "object": "0x75-u-",
    "object": "0x76-v-",
    "object": "0x77-w-",
    "object": "0x78-x-",
    "object": "0x79-y-",
    "object": "0x7A-z-",
    "object": "0x7B-{-",
    "object": "0x7C-|-",
    "object": "0x7D-}-",
    "object": "0x7E-~-",
    "object": "0x7F-\u007f-",
    "object": "Héroique",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.1",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.10",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.11",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.12",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.13",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.14",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.15",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.16",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.17",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.18",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.19",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.2",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.20",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.21",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.22",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.23",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.24",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.25",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.26",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.27",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.28",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.29",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.3",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.30",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.31",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.32",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.33",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.34",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.35",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.36",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.37",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.38",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.39",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.4",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.40",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.41",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.42",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.43",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.44",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.45",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.46",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.47",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.48",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.5",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.6",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.7",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.8",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.9",
    "object": "_multipart_part-00000.2~udBbNs7YuU7r5nNFIPsUZH1C-IEe9ep.meta",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.1",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.10",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.11",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.12",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.13",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.14",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.15",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.16",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.17",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.18",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.19",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.2",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.20",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.21",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.22",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.23",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.24",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.25",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.26",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.27",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.28",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.29",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.3",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.30",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.31",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.32",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.33",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.34",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.35",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.36",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.37",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.38",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.39",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.4",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.40",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.41",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.42",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.43",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.44",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.45",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.46",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.47",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.48",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.49",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.5",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.50",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.51",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.52",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.53",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.54",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.55",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.56",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.57",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.58",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.59",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.6",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.60",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.61",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.62",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.63",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.64",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.65",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.66",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.67",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.68",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.7",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.8",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.9",
    "object": "_multipart_part-00001.2~HSeCbhfJi7RQMnu5uqu4g7xiTFpz7Fy.meta",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.1",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.10",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.11",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.12",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.13",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.14",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.15",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.16",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.17",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.18",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.19",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.2",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.20",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.21",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.22",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.23",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.24",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.25",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.26",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.27",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.28",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.29",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.3",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.30",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.31",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.32",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.33",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.34",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.35",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.36",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.37",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.38",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.39",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.4",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.40",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.41",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.42",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.43",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.44",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.45",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.46",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.47",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.48",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.49",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.5",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.50",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.51",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.52",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.53",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.54",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.55",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.56",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.57",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.58",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.59",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.6",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.60",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.61",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.62",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.63",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.64",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.65",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.66",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.67",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.68",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.69",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.7",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.8",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.9",
    "object": "_multipart_part-00002.2~Auv9Tz2YH-_JIZVzp6Gb9tclOy7PY9k.meta",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.1",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.10",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.11",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.12",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.13",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.14",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.15",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.16",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.17",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.18",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.19",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.2",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.20",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.21",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.22",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.23",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.24",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.25",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.26",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.27",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.28",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.29",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.3",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.30",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.31",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.32",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.33",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.34",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.35",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.36",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.37",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.38",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.39",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.4",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.40",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.41",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.42",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.43",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.44",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.45",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.46",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.47",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.48",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.49",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.5",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.6",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.7",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.8",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.9",
    "object": "_multipart_part-00003.2~DtBaZuzLuHpUZSsV_OQo8oN-lgR2GjS.meta",
    "object": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" 
}
{
    "existing_header": {
        "usage": {
            "rgw.main": {
                "size": 1226840320,
                "size_actual": 1227358208,
                "size_utilized": 1226840320,
                "size_kb": 1198087,
                "size_kb_actual": 1198592,
                "size_kb_utilized": 1198087,
                "num_objects": 362
            },
            "rgw.multimeta": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 4
            }
        }
    },
    "calculated_header": {
        "usage": {
            "rgw.main": {
                "size": 1226840320,
                "size_actual": 1227358208,
                "size_utilized": 1226840320,
                "size_kb": 1198087,
                "size_kb_actual": 1198592,
                "size_kb_utilized": 1198087,
                "num_objects": 362
            },
            "rgw.multimeta": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 4
            }
        }
    }
}

spinn3r.json.xml View (34.1 KB) Yoann Moulin, 03/01/2018 01:09 PM

cephprod-rgw-log.tgz (180 KB) Yoann Moulin, 03/01/2018 03:09 PM

cephprod-rgw-iccluster004.log.1.gz (29.8 KB) Yoann Moulin, 03/07/2018 07:12 AM

History

#1 Updated by Yoann Moulin about 6 years ago

some additional logs fronm radosgw

#2 Updated by Yoann Moulin about 6 years ago

I found plenty of "execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json" in radosgw logs

2018-03-01 11:03:48.436376 7feac1218700 1 civetweb: 0x55cad4f93000: 127.0.0.1 - - [01/Mar/2018:11:03:48 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:14.757214 7feab7a05700 1 civetweb: 0x55cad4ffc000: 127.0.0.1 - - [01/Mar/2018:11:08:14 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:14.775363 7fead5a41700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:08:18.647447 7feab8206700 1 civetweb: 0x55cad4ff7000: 127.0.0.1 - - [01/Mar/2018:11:08:18 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:18.663724 7feaca22a700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:08:26.920768 7feaa81e6700 1 civetweb: 0x55cad509b000: 127.0.0.1 - - [01/Mar/2018:11:08:26 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:26.932970 7feac1a19700 1 civetweb: 0x55cad4f8e000: 127.0.0.1 - - [01/Mar/2018:11:08:26 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:08:26.950050 7feaae9f3700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.141228 7feac7a25700 1 civetweb: 0x55cad4f4e000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.157647 7feaa99e9700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.387137 7feabfa15700 1 civetweb: 0x55cad4fac000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.401582 7fead1238700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.625722 7feabb20c700 1 civetweb: 0x55cad4fd9000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.636035 7feaa69e3700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:37.848721 7fead2a3b700 1 civetweb: 0x55cad4ed4000: 127.0.0.1 - - [01/Mar/2018:11:09:37 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:37.863566 7feaae1f2700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.087841 7feacca2f700 1 civetweb: 0x55cad4f17000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.104343 7feac6222700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.337346 7feac8226700 1 civetweb: 0x55cad4f49000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.352437 7fead0a37700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json
2018-03-01 11:09:38.580932 7feaa89e7700 1 civetweb: 0x55cad5096000: 127.0.0.1 - - [01/Mar/2018:11:09:38 +0100] "GET /spinn3r.json/?location HTTP/1.0" 1 0 - -
2018-03-01 11:09:38.595777 7feaab9ed700 0 execute can't find bucket IAM POLICY attr bucket_name = spinn3r.json

#3 Updated by Orit Wasserman about 6 years ago

  • Assignee set to Pritha Srivastava

#4 Updated by Nathan Cutler about 6 years ago

  • Project changed from Ceph to rgw

#5 Updated by Yehuda Sadeh about 6 years ago

  • Priority changed from Normal to High

#6 Updated by Pritha Srivastava about 6 years ago

Can you please increase the logging level to 20 and then attach logs.

#7 Updated by Yoann Moulin about 6 years ago

ceph-post-file: 26c152cf-1fa8-4ef8-9bb8-1621bfcd43bc

#8 Updated by Pritha Srivastava about 6 years ago

Did you attach any file? Sorry I didnt get it.

#9 Updated by Pritha Srivastava about 6 years ago

Hi Yoann, Can you please attach the log files here. Not sure how to get the files that you uploaded using ceph-post-file. Or if you know how to get them, please let me know.

#10 Updated by Yoann Moulin about 6 years ago

here the log file

#11 Updated by Pritha Srivastava about 6 years ago

Hi Yoann, I checked all the log files including the latest one, but I dont see any errors there. I also tried to reproduce the issue by uploading object with utf-8 chars in its name, but i wasnt able to reproduce the issue. If you can give me the exact steps to repro the issue then it will help.

#12 Updated by Yoann Moulin about 6 years ago

That's really strange, I have a dataset on a rbd device on the same cluster, I just did.

rclone mkdir account:dataset
rclone copy dataset/ account:datatset/

Everything went well, but at the end of the copy, I got this error and weren't able to list the content of the dataset due to this error. I was able to list files separately. After those errors, I decided to remove the bucket without success, I have been able to delete files in the bucket one by one but it still impossible to remove this bucket.

I have no idea what happened, I'll install a test cluster and try to upload this dataset again and see if this issue happens again.

#13 Updated by Pritha Srivastava about 6 years ago

The last log file that you had attached with increased log levels, had two operations:

1. Delete Bucket, which failed with a Directory Not Empty error, because the objects were not deleted from it
2018-03-06 17:22:27.488380 7f2bd9b33700 2 req 1:0.011615:s3:DELETE /spinn3r.json:delete_bucket:verifying op params
2018-03-06 17:22:27.488384 7f2bd9b33700 2 req 1:0.011617:s3:DELETE /spinn3r.json:delete_bucket:pre-executing
2018-03-06 17:22:27.488387 7f2bd9b33700 2 req 1:0.011622:s3:DELETE /spinn3r.json:delete_bucket:executing
...
2018-03-06 17:22:27.510499 7f2bd9b33700 2 req 1:0.033733:s3:DELETE /spinn3r.json:delete_bucket:completing
2018-03-06 17:22:27.510625 7f2bd9b33700 2 req 1:0.033859:s3:DELETE /spinn3r.json:delete_bucket:op status=-39
2018-03-06 17:22:27.510629 7f2bd9b33700 2 req 1:0.033864:s3:DELETE /spinn3r.json:delete_bucket:http status=409
2018-03-06 17:22:27.510631 7f2bd9b33700 1 ====== req done req=0x7f2bd9b2d1f0 op status=-39 http_status=409 ======

2. Get Bucket, which succeeded
2018-03-06 17:22:58.499629 7f2bd2b25700 2 req 5:0.000608:s3:GET /:list_bucket:verifying op params
2018-03-06 17:22:58.499632 7f2bd2b25700 2 req 5:0.000610:s3:GET /:list_bucket:pre-executing
2018-03-06 17:22:58.499634 7f2bd2b25700 2 req 5:0.000613:s3:GET /:list_bucket:executing
...
2018-03-06 17:22:58.510320 7f2bd2b25700 2 req 5:0.011299:s3:GET /:list_bucket:http status=200
2018-03-06 17:22:58.510403 7f2bd2b25700 1 ====== req done req=0x7f2bd2b1f1f0 op status=0 http_status=200 ======

I didnt see any error in the bucket list operation. The Delete bucket operation failed because the bucket still had objects in it. But apart from these two operations, none of the other requests (like deleting every object individually), was logged. So it is difficult to conclude why the bucket wasn't empty.

#14 Updated by Yoann Moulin about 6 years ago

I just uploaded files of the bucket on a new fresh ceph cluster and everything is fine, I can list bucket files without any issue.

the next step for me is, How I can delete this bucket to be able to re upload it with the same name.

Yoann

#15 Updated by Yoann Moulin about 6 years ago

I didn't think to use --purge-objects before :

  radosgw-admin bucket rm --purge-objects --bucket=<bucket>

With this option, I have been able to rm the bucket, recreate it and upload files again, I didn't get any error and I'm able to list files. I have no idea what happened but now it works for me, we can close this issue.

Thanks for your help

Yoann

#16 Updated by Yehuda Sadeh about 6 years ago

I had a similar issue in a dev branch. The problem was that for some reason one of the properties in bucket list operation got extra junk. The culprit was a bit tricky to identify, but it ended up being a racing bucket list and dir_suggest, and the dir_suggest was feeding in bad data. The reason for it was that in my branch I removed the extra null character of the etag field, and dir_suggest accessed that xattr using bufferlist.c_str(). It could be that we have the same issue with some other field (e.g., content_type).

#17 Updated by Orit Wasserman almost 6 years ago

  • Status changed from New to Closed

Also available in: Atom PDF