Project

General

Profile

Bug #49206

rgw: deleting a bucket results in infinite loop

Added by Jeegn Chen almost 2 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific octopus nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When deleting a bucket with an incomplete multipart upload that
has about 2000 parts uploaded, we noticed an infinite loop, which
stopped s3cmd from deleting the bucket forever.
Per check on the source code, when the bucket index was sharded (for example 128
shards), the original logic in
RGWRados::cls_bucket_list_unordered() did not calculate
the bucket shard ID correctly when the index key of a data
part was taken as the marker.

The issue is not necessarily reproduced each time. It will depend
on the key of the object. To reproduce it in 128-shard bucket,
we use 334 as the key for the incomplete multipart upload,
which will be located in Shard 127 (known by experiment). In this
setup, the original logic will usually come out a shard ID smaller
than 127 (since 127 is the largest one) from the marker and
thus a circle is constructed, which results in an infinite loop.

PS: Sometimes the bucket ID calculation may incorrectly going forward
instead of backward. Thus, the check logic may skip some shards,
which may have regular keys. In such scenarios, some non-empty buckets may
be deleted by accident.


Related issues

Copied to rgw - Backport #51329: pacific: rgw: deleting a bucket results in infinite loop Resolved
Copied to rgw - Backport #51330: octopus: rgw: deleting a bucket results in infinite loop Resolved
Copied to rgw - Backport #52827: nautilus: rgw: deleting a bucket results in infinite loop Rejected

History

#2 Updated by Casey Bodley almost 2 years ago

  • Status changed from New to Fix Under Review
  • Backport set to pacific octopus nautilus

#3 Updated by J. Eric Ivancich over 1 year ago

  • Pull request ID set to 39358

#4 Updated by J. Eric Ivancich over 1 year ago

  • Assignee set to Jeegn Chen

#5 Updated by J. Eric Ivancich over 1 year ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from pacific octopus nautilus to pacific octopus

#6 Updated by Backport Bot over 1 year ago

  • Copied to Backport #51329: pacific: rgw: deleting a bucket results in infinite loop added

#7 Updated by Backport Bot over 1 year ago

  • Copied to Backport #51330: octopus: rgw: deleting a bucket results in infinite loop added

#8 Updated by Loïc Dachary about 1 year ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

#9 Updated by Mykola Golub about 1 year ago

  • Backport changed from pacific octopus to pacific octopus nautilus

#10 Updated by Mykola Golub about 1 year ago

  • Status changed from Resolved to Pending Backport

#11 Updated by Backport Bot about 1 year ago

  • Copied to Backport #52827: nautilus: rgw: deleting a bucket results in infinite loop added

#12 Updated by Backport Bot 4 months ago

  • Tags set to backport_processed

#13 Updated by Konstantin Shalygin 4 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF