Project

General

Profile

Actions

Feature #41051

closed

rgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performance

Added by J. Eric Ivancich over 4 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus
Reviewed:
Affected Versions:
Pull request ID:

Description

This is a suggestion from Paul Emmerich on the ceph-users mailing list....

Common prefixes could filtered in the rgw class on the OSD instead of in radosgw

Consider a bucket with 100 folders with 1000 objects in each and only one shard

/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000

Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.

How much data will be transfered between the OSDs and radosgw for this request?
How many omap entries does the OSD scan?

radosgw will ask the (single) index object to list the first 1000 objects. It'll
return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000

radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.

So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.

This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?

IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.


Related issues 1 (0 open1 closed)

Copied to rgw - Backport #43823: nautilus: rgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performanceRejectedJ. Eric IvancichActions
Actions #1

Updated by J. Eric Ivancich over 4 years ago

  • Assignee set to J. Eric Ivancich
Actions #2

Updated by Simon Leinen over 4 years ago

Should this issue link to PR #30272?

Actions #3

Updated by Simon Leinen over 4 years ago

Simon Leinen wrote:

Should this issue link to PR #30272?

...or maybe folded into issue #41734

Actions #4

Updated by J. Eric Ivancich over 4 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 30272
Actions #5

Updated by J. Eric Ivancich over 4 years ago

Thanks, Simon. That was the original intention.

Actions #6

Updated by J. Eric Ivancich over 4 years ago

  • Status changed from Fix Under Review to 7
Actions #7

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 7 to Fix Under Review
Actions #8

Updated by Casey Bodley about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #10

Updated by Nathan Cutler about 4 years ago

  • Status changed from Pending Backport to Fix Under Review
Actions #11

Updated by J. Eric Ivancich about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #12

Updated by Nathan Cutler about 4 years ago

  • Copied to Backport #43823: nautilus: rgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performance added
Actions #13

Updated by J. Eric Ivancich about 3 years ago

  • Status changed from Pending Backport to Resolved

Backport rejected.

Actions #14

Updated by Jiayu Sun almost 2 years ago

I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?

Actions #15

Updated by Casey Bodley almost 2 years ago

Jiayu Sun wrote:

I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?

the backport was rejected because the nautilus release is no longer maintained. we'd recommend upgrading to a supported ceph release for this fix

Actions #16

Updated by Jiayu Sun almost 2 years ago

Casey Bodley wrote:

Jiayu Sun wrote:

I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?

the backport was rejected because the nautilus release is no longer maintained. we'd recommend upgrading to a supported ceph release for this fix

I hvae upgraded to latest pacific release. does this mean that this patch have already been integrated in pacific version?

Actions #17

Updated by Paul Emmerich almost 2 years ago

Yes, the fix is in all builds of Pacific.

Actions

Also available in: Atom PDF