Bug #62646: RGW Swift info not providing max_deletes_per_request in bulk_delete section - rgw - Ceph

Actions

Copy link

Bug #62646

closed

RGW Swift info not providing max_deletes_per_request in bulk_delete section

Added by Christian Rohmann 8 months ago. Updated 6 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Shreyansh Sancheti

Target version:

% Done:

Source:

Tags:

swift low-hanging-fruit

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

54050

Crash signature (v1):

Crash signature (v2):

Description

RGW implements the info endpoint for the Swift protocol - this was added with https://github.com/ceph/ceph/commit/277079da3c57f61221254f266b698ab3962ef87c

Currently the Ceph RADOSGW presents a key bulk_delete with no subkeys or values:

{
  "bulk_delete": {},
  "container_quotas": {},
  "swift": {
    "max_file_size": 5368709120,
    "container_listing_limit": 10000,
    "version": "17.2.6",
    "policies": [
      {
        "default": true,
        "name": "default-placement" 
      }
    ],
    "max_object_name_size": 1024,
    "strict_cors_mode": true,
    "max_container_name_length": 255
  },
  "tempurl": {
    "methods": [
      "GET",
      "HEAD",
      "PUT",
      "POST",
      "DELETE" 
    ]
  },
  "slo": {
    "max_manifest_segments": 1000
  },
  "account_quotas": {},
  "staticweb": {},
  "tempauth": {
    "account_acls": true
  }
}

If you compare this to what OpenStack Swift implements at https://github.com/openstack/swift/blame/191b627b856bc2e8eaa4049521e3476bf29ed40c/swift/common/middleware/bulk.py#L773 or documents at https://github.com/openstack/python-swiftclient/blob/54fbfa8bc9c10feb48fbce843d6ba6fcf6533ece/doc/source/service-api.rst#capabilities (at least) the keys max_deletes_per_request and max_failed_deletes are missing.

While Swift clients might(!) enable bulk_delete operations they are unable to determine how many deletes can actually be issued and have to fall back to some default, sane or not.

Presenting these limits correctly to Swift clients should be rather easy but potentially speed up bulk_deletes.

Actions

Copy link

Updated by Casey Bodley 7 months ago

it looks like the config option controls rgw_delete_multi_obj_max_num, so we can add that value to the swift info response. i don't know that the max_failed_deletes part is implemented

Actions

Copy link

Updated by Casey Bodley 7 months ago

Tags set to swift low-hanging-fruit

Actions

Copy link

Updated by Christian Rohmann 7 months ago

Shreyansh Sancheti wrote:

Need info on what command output is pasted in the description.

That's the Swift info endpoint, so https://$rgw_hostname/swift/info
(in case the default path of /swift is set in RGW to provide the Swift protocol)

Actions

Copy link

Updated by Shreyansh Sancheti 7 months ago

Christian Rohmann wrote:

Shreyansh Sancheti wrote:

Need info on what command output is pasted in the description.

That's the Swift info endpoint, so https://$rgw_hostname/swift/info
(in case the default path of /swift is set in RGW to provide the Swift protocol)

Okay got it, so the regular behaviour is we should see some default entries in the bulk_delete or we have to set some options and then when we do swift info it should show?

Actions

Copy link

Updated by Christian Rohmann 7 months ago

Shreyansh Sancheti wrote:

That's the Swift info endpoint, so https://$rgw_hostname/swift/info
(in case the default path of /swift is set in RGW to provide the Swift protocol)

Okay got it, so the regular behaviour is we should see some default entries in the bulk_delete or we have to set some options and then when we do swift info it should show?

There are two parameters which I identified that are missing:

1) max_deletes_per_request
2) max_failed_deletes

For 1) it's either a sensible static value for what Ceph RGW supports for max_deletes_per_request out of the box or as Casey said in https://tracker.ceph.com/issues/62646#note-1 simply the value of the already existing config key rgw_delete_multi_obj_max_num.

For 2) the case is different as that might even be a missing feature as there is no configurable value in the RGW config.
But if you read https://github.com/openstack/swift/blob/d31a54a65c186550a99bb985f056c071690a3939/etc/proxy-server.conf-sample#L25 it seems this is not a public value anyways. If you search through the Swift code (https://github.com/search?q=repo%3Aopenstack%2Fswift%20max_failed_deletes&type=code) it seems it has a default value of 1000, but again, that is not even publicly shown. So fixing 1) is much more important to allow all clients to issue bulk requests properly.

Actions

Copy link