Bug #62006
opensome objects has size 0 on listing, but on head and get - full contents
0%
Description
hello!
We have a problem with some amount of objects (not all), when listed with any s3 tool - they show size 0, while HEAD method return correct non zero size. The object can be also downloaded and the content is correct.
Wrong size on listing
DEBUG: Sending request method_string='GET', uri='/bucket/data', headers={'x-amz-date': '20230713T130329Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=DFG1MLVAJ57BEKLEH485/20230713/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=641f640b742a612c905df8800f8f5969bab8f36aaef1d617f035b530c1e33de2', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://s3.city.com:8080#2)
DEBUG: Response:
{'data': b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http:'
b'//s3.amazonaws.com/doc/2006-03-01/"><Tenant>sdffdsfsdfdsfgcdcscdscsd'
b'624e56b5</Tenant><Name>tenanttenanttenanttenan</Name><Prefix>/data</Prefix><MaxKeys>1000</MaxKeys><'
b'Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Ke'
b'y>data</Key><LastModified>2023'
b'-06-21T11:41:42.362Z</LastModified><ETag>"d41d8cd98f00b204e9800'
b'998ecf8427e"</ETag><Size>0</Size><StorageClass>STANDARD</Storag'
b'eClass><Owner><ID>sdffdsfsdfdsfgcdcscdscsd$sdffdsfsdfdsfgcdcscdscsd'
b'c412202624e56b5</ID><DisplayName>sdffdsfsdfdsfgcdcscdscsd</DisplayName></Own'
b'er><RgwxTag>_SMi2f9QbphDuyptioCMid4mZm5Z3Pz8</RgwxTag><Type>Normal</'
b'Type></Contents><Marker></Marker></ListBucketResult>',
'headers': {'content-type': 'application/xml',
'date': 'Thu, 13 Jul 2023 13:03:30 GMT',
'transfer-encoding': 'chunked',
'x-amz-request-id': 'tx00000c4800f3654d061a7-0064aff622-75fb432-default'},
'reason': 'OK',
'status': 200}
Correct on info
DEBUG: Sending request method_string='HEAD', uri='/bucket/data', headers={'x-amz-date': '20230713T130334Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=DFG1MLVAJ57BEKLEH485/20230713/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=a2c9dc0abd582e702bf55b6bdf3151e3726d7b978711f692003b69e54a1518fc', 'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://s3.city.com:8080#2)
DEBUG: Response:
{'data': b'',
'headers': {'accept-ranges': 'bytes',
'content-length': '276229216',
'content-type': 'application/octet-stream',
'date': 'Thu, 13 Jul 2023 13:03:34 GMT',
'etag': '"016084345f7df2c4fcd229d3bcdaf68c"',
'last-modified': 'Wed, 21 Jun 2023 11:41:42 GMT',
'rgwx-mtime': '1687347702.362601263',
'rgwx-obj-pg-ver': '8953051',
'x-amz-request-id': 'tx00000907002f211e50fa6-0064aff626-6c98ff2-default',
'x-object-manifest': '/data/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f63617265726f6f6d2f63617265726f6f6d2d7068702f5f75706c6f6164732f35323331393335312d393234632d343839652d623238652d3039616166623061366631352f6461746186fa526a60e7b62ecffb6fef3109c2b16fba9ebce9e2e3f063a9c6a4090deed0da39a3ee5e6b4b0d3255bfef95601890afd80709',
'x-rgw-object-type': 'Normal'},
'reason': 'OK',
'status': 200}
s3://sdffdsfsdfdsfgcdcscdscsd:bucket/data (object):
File size: 276229216
Last mod: Wed, 21 Jun 2023 11:41:42 GMT
MIME type: application/octet-stream
Storage: STANDARD
MD5 sum: 016084345f7df2c4fcd229d3bcdaf68c
SSE: none
Updated by Casey Bodley 10 months ago
i don't think we've seen issues like this before, where the bucket index is getting the wrong size. can you try to narrow down how to reproduce this?
Updated by Pawel Stefanski 10 months ago
Casey Bodley wrote:
i don't think we've seen issues like this before, where the bucket index is getting the wrong size. can you try to narrow down how to reproduce this?
yes, this is pretty weird, we have very similar setup in the other regions, at the same version and even the same customer with almost the same data (or at least the same use case and structure) and there everything is fine.
Will enabled more debug and try to catch those GET reqs in logs.
Updated by Pawel Stefanski 10 months ago
The RGW log of this "ls" request is here.
https://gist.github.com/pejotes/43c8a14929eb523004575f8e56a97390
It's exactly the same when I use radosgw-admin object stat method on the bucket and object. The response has "size": 0, and "obj_size": 0, and "head_size": 0.
The bucket was created with 16 shards from the beginning and it wasn't resharded or manipulated (according to my knowledge).
I've tried to fix this with
radosgw-admin bucket check --bucket 'tenant/bucket' --check-objects --fix
but no luck.