Project

General

Profile

Bug #58190

Large RGW GC queue might prevent OSD from starting

Added by Igor Fedotov about 2 months ago. Updated 6 days ago.

Status:
Pending Backport
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
backport_processed
Backport:
quincy, pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

It looks like rgw_gc_queue_list_entries might cause HDD-based OSD to load the queue for more than half an hour.
Which causes OSD being marked as down shortly after restart.
The root cause is that the queue (or rather some entry in it) takes approx. 60 MB and rgw_gc_queue_list_entries reads it through pretty inefficient CLS queue_list_entries using 1K (sic!) read chunks. Which in turn permits 7-8 reads from a spinning drive.
Relevant OSD log is attached.

TextFile5.txt View - very long running rgw_gc_queue_list_entries() (the beginning and a middle part) (29.6 KB) Igor Fedotov, 12/06/2022 04:25 PM


Related issues

Related to rgw - Bug #53585: RGW Garbage collector leads to slow ops and osd down when removing large object New
Copied to rgw - Backport #58579: pacific: Large RGW GC queue might prevent OSD from starting In Progress
Copied to rgw - Backport #58580: quincy: Large RGW GC queue might prevent OSD from starting In Progress

History

#1 Updated by Igor Fedotov about 2 months ago

  • Severity changed from 3 - minor to 2 - major

#2 Updated by Matt Benjamin about 2 months ago

I think this could also be interacting with the issue being addressed here:

https://github.com/ceph/ceph/pull/48839

Matt

#3 Updated by Igor Fedotov about 2 months ago

Matt Benjamin wrote:

I think this could also be interacting with the issue being addressed here:

https://github.com/ceph/ceph/pull/48839

Matt

Matt, IIUC your point is that queue_list_entries might take too long time due to large bufferlist and resulting copying overhead, is that true?

IMO that's not the case for this ticket - the issue is that reading 60MB entry using 1K chunks from spinning drive is absolutely inappropriate - each read takes approx. 150ms hence reading the full entry takes more than half an hour!!! And even worse - for unknown reason this stalls all other exchange with an OSD too...

Migrating the pool to SSD drives resolved the issue - so it's [disk] reading performance which is crucial...

#4 Updated by Matt Benjamin about 2 months ago

That's certainly very interesting and, hopefully, trivially tunable?

Matt

#5 Updated by Igor Fedotov about 2 months ago

  • Status changed from New to Fix Under Review
  • Backport set to quincy, pacific
  • Pull request ID set to 49313

#6 Updated by Igor Fedotov about 2 months ago

Matt Benjamin wrote:

That's certainly very interesting and, hopefully, trivially tunable?

Matt

Hopefully yes. See the PR, please..

#7 Updated by Igor Fedotov about 2 months ago

  • Related to Bug #53585: RGW Garbage collector leads to slow ops and osd down when removing large object added

#8 Updated by Matt Benjamin about 2 months ago

@igor, any chance you can come to an RGW Bug Scrub (Thursdays, 10am us eastern)?

Matt

#9 Updated by Igor Fedotov about 2 months ago

Matt Benjamin wrote:

@igor, any chance you can come to an RGW Bug Scrub (Thursdays, 10am us eastern)?

Matt

I think so.

#10 Updated by Casey Bodley 12 days ago

  • Priority changed from Normal to High

#11 Updated by Casey Bodley 6 days ago

  • Status changed from Fix Under Review to Pending Backport

#12 Updated by Backport Bot 6 days ago

  • Copied to Backport #58579: pacific: Large RGW GC queue might prevent OSD from starting added

#13 Updated by Backport Bot 6 days ago

  • Copied to Backport #58580: quincy: Large RGW GC queue might prevent OSD from starting added

#14 Updated by Backport Bot 6 days ago

  • Tags set to backport_processed

Also available in: Atom PDF