Bug #58190
closed
Large RGW GC queue might prevent OSD from starting
Added by Igor Fedotov over 1 year ago.
Updated 12 months ago.
Description
It looks like rgw_gc_queue_list_entries might cause HDD-based OSD to load the queue for more than half an hour.
Which causes OSD being marked as down shortly after restart.
The root cause is that the queue (or rather some entry in it) takes approx. 60 MB and rgw_gc_queue_list_entries reads it through pretty inefficient CLS queue_list_entries using 1K (sic!) read chunks. Which in turn permits 7-8 reads from a spinning drive.
Relevant OSD log is attached.
Files
|
TextFile5.txt (29.6 KB)
TextFile5.txt |
very long running rgw_gc_queue_list_entries() (the beginning and a middle part) |
Igor Fedotov, 12/06/2022 04:25 PM
|
|
- Severity changed from 3 - minor to 2 - major
Matt Benjamin wrote:
I think this could also be interacting with the issue being addressed here:
https://github.com/ceph/ceph/pull/48839
Matt
Matt, IIUC your point is that queue_list_entries might take too long time due to large bufferlist and resulting copying overhead, is that true?
IMO that's not the case for this ticket - the issue is that reading 60MB entry using 1K chunks from spinning drive is absolutely inappropriate - each read takes approx. 150ms hence reading the full entry takes more than half an hour!!! And even worse - for unknown reason this stalls all other exchange with an OSD too...
Migrating the pool to SSD drives resolved the issue - so it's [disk] reading performance which is crucial...
That's certainly very interesting and, hopefully, trivially tunable?
Matt
- Status changed from New to Fix Under Review
- Backport set to quincy, pacific
- Pull request ID set to 49313
Matt Benjamin wrote:
That's certainly very interesting and, hopefully, trivially tunable?
Matt
Hopefully yes. See the PR, please..
- Related to Bug #53585: RGW Garbage collector leads to slow ops and osd down when removing large object added
@Igor Gajowiak, any chance you can come to an RGW Bug Scrub (Thursdays, 10am us eastern)?
Matt
Matt Benjamin wrote:
@Igor Gajowiak, any chance you can come to an RGW Bug Scrub (Thursdays, 10am us eastern)?
Matt
I think so.
- Priority changed from Normal to High
- Status changed from Fix Under Review to Pending Backport
- Copied to Backport #58579: pacific: Large RGW GC queue might prevent OSD from starting added
- Copied to Backport #58580: quincy: Large RGW GC queue might prevent OSD from starting added
- Tags set to backport_processed
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF