Bug #58190: Large RGW GC queue might prevent OSD from starting - rgw - Ceph

Actions

Copy link

Bug #58190

closed

Large RGW GC queue might prevent OSD from starting

Added by Igor Fedotov over 1 year ago. Updated 11 months ago.

Status:

Resolved

Priority:

High

Assignee:

Target version:

% Done:

Source:

Community (dev)

Tags:

backport_processed

Backport:

quincy, pacific

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v16.2.10

ceph-qa-suite:

Pull request ID:

49313

Crash signature (v1):

Crash signature (v2):

Description

It looks like rgw_gc_queue_list_entries might cause HDD-based OSD to load the queue for more than half an hour.
Which causes OSD being marked as down shortly after restart.
The root cause is that the queue (or rather some entry in it) takes approx. 60 MB and rgw_gc_queue_list_entries reads it through pretty inefficient CLS queue_list_entries using 1K (sic!) read chunks. Which in turn permits 7-8 reads from a spinning drive.
Relevant OSD log is attached.

Files

TextFile5.txt (29.6 KB) TextFile5.txt

very long running rgw_gc_queue_list_entries() (the beginning and a middle part)

Igor Fedotov, 12/06/2022 04:25 PM

Related issues 3 (1 open — 2 closed)

Actions

Copy link

Updated by Igor Fedotov over 1 year ago

Severity changed from 3 - minor to 2 - major

Actions

Copy link

Updated by Matt Benjamin over 1 year ago

I think this could also be interacting with the issue being addressed here:

https://github.com/ceph/ceph/pull/48839

Matt

Actions

Copy link

Updated by Igor Fedotov over 1 year ago

Matt Benjamin wrote:

I think this could also be interacting with the issue being addressed here:

https://github.com/ceph/ceph/pull/48839

Matt

Matt, IIUC your point is that queue_list_entries might take too long time due to large bufferlist and resulting copying overhead, is that true?

IMO that's not the case for this ticket - the issue is that reading 60MB entry using 1K chunks from spinning drive is absolutely inappropriate - each read takes approx. 150ms hence reading the full entry takes more than half an hour!!! And even worse - for unknown reason this stalls all other exchange with an OSD too...

Migrating the pool to SSD drives resolved the issue - so it's [disk] reading performance which is crucial...

Actions

Copy link