Project

General

Profile

Actions
Copy link

Bug #63721

open

deleting bucket failed: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-27

Added by Mykola Golub 5 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
J. Eric Ivancich
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

One of our users cannot delete several buckets with radosgw-admin due to the error like below:

2023-11-28T07:48:32.486+0100 7f323910a840  0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-27
2023-11-28T07:48:32.486+0100 7f323910a840  0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -27 for :CENSOREDBUCKETNAME[abc69c9f-3a61-465b-aa4d-4110bf79d972.896755.1])

So it looks like when the osd's rgw_bucket_list returns RGWBIAdvanceAndRetryError this spacial error code is not handled correctly in the case of unordered list and just bubble up to the upper layer as an ordinary error.

The `radosgw-admin bucket list --allow-unordered` also fails for these buckets:

ERROR: store->list_objects(): 2023-11-29T12:45:44.952+0100 7ff772ff7840  0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-27
(27) File too large2023-11-29T12:45:44.952+0100 7ff772ff7840  0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -27 for :CENSOREDBUCKETNAME[abc69c9f-3a61-465b-aa4d-4110bf79d972.896755.1])

The buckets use versioning and I suppose this is the source of so large number of "not visible" entries that it has to return RGWBIAdvanceAndRetryError.

Ceph version: 17.2.5

Actions
Copy link
 #1

Updated by Casey Bodley 5 months ago

  • Assignee set to J. Eric Ivancich
Actions
Copy link
 #2

Updated by J. Eric Ivancich 5 months ago

Are you able to share the output of `radosgw-admin bi list --bucket=<bucket_name> --max-entries=9999999` for the bucket that you're not able to do an unordered list with `radosgw-admin bucket list --allow-unordered ...`?

I'm trying to test this out, but unsure why you're having long sequences of undisplayable entries. One possible route might be lots of older versions for a given object if the listing did not include versions, but `radosgw-admin bucket list ...` does include versions.

Thanks!

Actions
Copy link
 #3

Updated by J. Eric Ivancich 4 months ago

  • Status changed from New to Need More Info
Actions
Copy link
 #4

Updated by Mykola Golub 4 months ago

  • Status changed from Need More Info to New

J. Eric Ivancich wrote:

Are you able to share the output of `radosgw-admin bi list --bucket=<bucket_name> --max-entries=9999999` for the bucket that you're not able to do an unordered list with `radosgw-admin bucket list --allow-unordered ...`?

As I wrote to you in slack, I put the file on teuthology host in my home directory (/home/trociny/*.bi.list.gz).

Actions
Copy link
 #5

Updated by Randall Wilson 3 months ago

We are seeing the same issue as well on several large buckets.

ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-27


0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -27 for :xxxx[5e9bc383-f7bd-4fd1-b607-1e563bfe0011.340335820.2])

Any type of rm job request dies immediately with the error including using the -allow-unordered flag. 17.2.7 quincy

Actions
Copy link

Also available in: Atom PDF