Project

General

Profile

Feature #22168

The RGW Admin OPS is missing the ability to filter for e.g. buckets and users

Added by Volker Theile over 3 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

Currently it is not possible to use the RGW Admin OPS API in scenarious with a large number of user and buckets. The problem in general is that the API does not provide the ability to query for users and buckets using filter options. This is really important if you want to use the API in an UI like openATTIC.

Because it is not able to filter the requested data via Admin OPS API we need to emulate this behaviour in the UI. Currently this is done the following way for buckets:

  • Get a list of ALL bucket names via GET /{admin}/bucket
  • Loop through that list of names and collect the full bucket information for each of them via GET /{admin}/bucket/bucket=xxx. This is necessary if you filter by owner or any additional field.
  • Now apply the filter settings from the UI datatable, e.g. filter by bucket name or owner. Finally apply the page and pageSize filter options to return the requested subset of the buckets to be displayed.

The JS code can be found at https://bitbucket.org/openattic/openattic/src/ad662cb95faed7ce410075f2c1f268466a2c0fd2/webui/app/components/ceph-rgw/shared/ceph-rgw-bucket.service.js?at=master&fileviewer=file-view-default#ceph-rgw-bucket.service.js-70.

Think about how long this requires if your RGW has about 100,000 buckets and you filter for an owner.

Because of that the RGW Admin OPS API requires the ability to apply filter options like this:

page The number of the requested page
pageSize A numeric value indicating the page size
searchfield The name of the field to search for
search The value to search for
sortfield The name of the field used for sorting
sortorder ASC or DESC

The result should return the requested subset plus the information how much buckets/users/xxx exists in total, e.g.

/{admin}/buckets?sortfield=uid&page=1&pageSize=10&search=&searchfield=uid

{
count: 18
result: [array of 10 bucket infos]
}

/{admin}/buckets?sortfield=uid&page=2&pageSize=10&search=&searchfield=uid

{
count: 18
result: [array of 8 bucket infos (starting from 11-18)]
}


Related issues

Related to mgr - Bug #44667: mgr/dashboard: In Object Gateway,Users and Buckets auto refresh or reload is not working Resolved
Related to mgr - Bug #40753: mgr/dashboard: Perform scalability tests with large amounts of RGW buckets Resolved
Related to mgr - Tasks #40766: mgr/dashboard: Perform scalability tests with large amounts of RGW users New 07/12/2019

History

#1 Updated by Casey Bodley about 2 years ago

note that 'GET /admin/bucket' can take a uid=foo to filter on a single user. this is fast because the user itself stores its list of buckets, so doesn't require filtering the global list of all buckets

in general though, these APIs are just listing rados objects in a pool, so we're limited by what the librados interfaces support - basically just 'give me the next N names, starting from name foo'

the filtering is easy for radosgw to do locally. but you won't be able to customize sort orders, start a listing at some arbitrary 'page 6', or know the total number of entries without someone having to read all entries into memory

#2 Updated by Lenz Grimmer 11 months ago

  • Related to Bug #44667: mgr/dashboard: In Object Gateway,Users and Buckets auto refresh or reload is not working added

#3 Updated by Volker Theile 11 months ago

  • Related to Bug #43869: mgr/dashboard: Teuthology test failure "test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest)" added

#4 Updated by Volker Theile 11 months ago

  • Related to deleted (Bug #43869: mgr/dashboard: Teuthology test failure "test_pwd_expiration_date_update (tasks.mgr.dashboard.test_user.UserTest)")

#5 Updated by Lenz Grimmer 6 months ago

  • Related to Bug #40753: mgr/dashboard: Perform scalability tests with large amounts of RGW buckets added

#6 Updated by Lenz Grimmer 6 months ago

  • Related to Tasks #40766: mgr/dashboard: Perform scalability tests with large amounts of RGW users added

Also available in: Atom PDF