Project

General

Profile

Actions

Bug #65436

open

Getting Object Crashing radosgw services

Added by Reid Guyett 19 days ago. Updated 12 days ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello,

We are seeing crashes when users are trying to get a specific file.

2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj get_obj_state: rctx=0x7f6fb47f6ac0 obj=<bucketname>:.cache/<112 characters>.jpg/<33 characters> state=0x7f698418d728 s->prefetch_data=1
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj get_obj_state: rctx=0x7f6fb47f6ac0 obj=<bucketname>:.cache/<112 characters>.jpg/<33 characters> state=0x7f698418d728 s->prefetch_data=1
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.olh.idtag
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.olh.info
2024-04-11T11:02:38.174+0000 7f6bbf6ee700 20 req 1201706618685104296 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.olh.ver
2024-04-11T11:02:38.178+0000 7f6bbf6ee700 -1 *** Caught signal (Aborted) **

This is reproducible by on this specific object:

$ s3cmd -c s3 get s3://<bucket>/.cache/<112 characters>.jpg/<33 characters> 
download: 's3://<bucket>/.cache/<112 characters>.jpg/<33 characters>' -> './<33 characters>'  [1 of 1]
ERROR: Error parsing xml: mismatched tag: line 6, column 2
ERROR: b'<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'
ERROR: Download of './<33 characters>' failed (Reason: 502 (Bad Gateway))
ERROR: S3 error: 502 (Bad Gateway)

We are running:
rgw on 17.2.5
rest is 17.2.7
on Debian 11


Files

radosgw crash details (scrubbed).log (15.7 KB) radosgw crash details (scrubbed).log Full rgw log Reid Guyett, 04/11/2024 05:10 PM
Actions #1

Updated by hoan nv 14 days ago

I have same issue. After some days, i found bug https://tracker.ceph.com/issues/61359
After upgrade to 17.2.7, this bug gone. But i should delete error file, i can't fix this file.

Actions #2

Updated by Casey Bodley 12 days ago

  • Status changed from New to Need More Info

After upgrade to 17.2.7, this bug gone

it sounds like this bug is fixed in later point release, can you please try to upgrade? we can't do anything to fix 17.2.5 specifically

Actions

Also available in: Atom PDF