Feature #41051

rgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performance

Added by Eric Ivancich 3 months ago. Updated about 1 month ago.

Need Review
Target version:
Start date:
Due date:
% Done:


Affected Versions:
Pull request ID:


This is a suggestion from Paul Emmerich on the ceph-users mailing list....

Common prefixes could filtered in the rgw class on the OSD instead of in radosgw

Consider a bucket with 100 folders with 1000 objects in each and only one shard

/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000

Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.

How much data will be transfered between the OSDs and radosgw for this request?
How many omap entries does the OSD scan?

radosgw will ask the (single) index object to list the first 1000 objects. It'll
return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000

radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.

So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.

This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?

IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.


#1 Updated by Eric Ivancich 3 months ago

  • Assignee set to Eric Ivancich

#2 Updated by Simon Leinen about 1 month ago

Should this issue link to PR #30272?

#3 Updated by Simon Leinen about 1 month ago

Simon Leinen wrote:

Should this issue link to PR #30272?

...or maybe folded into issue #41734

#4 Updated by Eric Ivancich about 1 month ago

  • Status changed from New to Need Review
  • Pull request ID set to 30272

#5 Updated by Eric Ivancich about 1 month ago

Thanks, Simon. That was the original intention.

Also available in: Atom PDF