Feature #41051
closedrgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performance
0%
Description
This is a suggestion from Paul Emmerich on the ceph-users mailing list....
Common prefixes could filtered in the rgw class on the OSD instead of in radosgw
Consider a bucket with 100 folders with 1000 objects in each and only one shard
/p1/1, /p1/2, ..., /p1/1000, /p2/1, /p2/2, ..., /p2/1000, ... /p100/1000
Now a user wants to list / with aggregating common prefixes on the delimiter / and wants up to 1000 results. So there'll be 100 results returned to the client: the common prefixes p1 to p100.
How much data will be transfered between the OSDs and radosgw for this request?
How many omap entries does the OSD scan?
radosgw will ask the (single) index object to list the first 1000 objects. It'll
return 1000 objects in a quite unhelpful way: /p1/1, /p1/2, ...., /p1/1000
radosgw will discard 999 of these and detect one common prefix and continue the iteration at /p1/\xFF to skip the remaining entries in /p1/ if there are any. The OSD will then return everything in /p2/ in that next request and so on.
So it'll internally list every single object in that bucket. That will be a problem for large buckets and having lots of shards doesn't help either.
This shouldn't be too hard to fix: add an option "aggregate prefixes" to the RGW class method and duplicate the fast-forward logic from radosgw in cls_rgw. It doesn't even need to change the response type or anything, it just needs to limit entries in common prefixes to one result. Is this a good idea or am I missing something?
IO would be reduced by a factor of 100 for that particular pathological case. I've unfortunately seen a real-world setup that I think hits a case like that.
Updated by Simon Leinen over 4 years ago
Updated by J. Eric Ivancich over 4 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 30272
Updated by J. Eric Ivancich over 4 years ago
Thanks, Simon. That was the original intention.
Updated by J. Eric Ivancich over 4 years ago
- Status changed from Fix Under Review to 7
Updated by Patrick Donnelly over 4 years ago
- Status changed from 7 to Fix Under Review
Updated by Casey Bodley over 4 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler over 4 years ago
- Status changed from Pending Backport to Fix Under Review
Updated by J. Eric Ivancich over 4 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler about 4 years ago
- Copied to Backport #43823: nautilus: rgw: push rgw bucket listing with prefixes and delimiters logic/filtering to the cls layer to improve performance added
Updated by J. Eric Ivancich about 3 years ago
- Status changed from Pending Backport to Resolved
Backport rejected.
Updated by Jiayu Sun almost 2 years ago
I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?
Updated by Casey Bodley almost 2 years ago
Jiayu Sun wrote:
I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?
the backport was rejected because the nautilus release is no longer maintained. we'd recommend upgrading to a supported ceph release for this fix
Updated by Jiayu Sun almost 2 years ago
Casey Bodley wrote:
Jiayu Sun wrote:
I'm facing the same performance issue. since this backport was rejected. Is there any other way to make performance better?
the backport was rejected because the nautilus release is no longer maintained. we'd recommend upgrading to a supported ceph release for this fix
I hvae upgraded to latest pacific release. does this mean that this patch have already been integrated in pacific version?
Updated by Paul Emmerich almost 2 years ago
Yes, the fix is in all builds of Pacific.