Project

General

Profile

Actions
Copy link

Bug #52662

closed

rgw: cannot delete bucket

Added by Paul Jurco over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
J. Eric Ivancich
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v15.2.14, Ceph - v15.2.15, Ceph - v16.2.6
ceph-qa-suite:
Pull request ID:
43862
Crash signature (v1):
Crash signature (v2):

Description

Hi,
Bucket deletion is not fixed by https://tracker.ceph.com/issues/50302
I have upgraded to 15.2.14 in order to be able to delete an old bucket stuck at:
2021-09-08T08:47:15.216+0300 7f96ddfe7080 0 abort_bucket_multiparts WARNING : aborted 34333 incomplete multipart uploads
2021-09-08T08:47:17.012+0300 7f96ddfe7080 0 abort_bucket_multiparts WARNING : aborted 34334 incomplete multipart uploads
2021-09-08T08:47:18.828+0300 7f96ddfe7080 0 abort_bucket_multiparts WARNING : aborted 34335 incomplete multipart uploads

But is seems it was not fixed.
The bug should have been fixed by:

https://docs.ceph.com/en/latest/releases/octopus/#v15-2-14-octopus

rgw: radoslist incomplete multipart parts marker (pr#40820, J. Eric Ivancich)

https://github.com/ceph/ceph/pull/40801

Bucket stats output:

"usage": {
        "rgw.main": {
            "size": 251428486576,
            "size_actual": 251428556800,
            "size_utilized": 251428486576,
            "size_kb": 245535632,
            "size_kb_actual": 245535700,
            "size_kb_utilized": 245535632,
            "num_objects": 47978
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 2
        }

Executed a check --fix --check-objects before.
Please have a look! Thank you!
Paul

Files

feed-v2-cannot-delete-bucket.txt.gz (666 KB) feed-v2-cannot-delete-bucket.txt.gz output of radosgw-admin bucket rm --bucket=feed-v2 --purge-objects --bypass-gc --debug-rgw=20 Paul Jurco, 09/29/2021 06:54 AM

Related issues 1 (0 open — 1 closed)

Copied to rgw - Backport #53213: octopus: rgw: cannot delete bucket ResolvedJ. Eric IvancichActions
Actions
Copy link
 #1

Updated by Daniel Gryniewicz over 2 years ago

  • Assignee set to J. Eric Ivancich
Actions
Copy link
 #2

Updated by J. Eric Ivancich over 2 years ago

  • Subject changed from Cannot delete bucket to rgw: cannot delete bucket

So the fix referred to only affects `radosgw-admin bucket radoslist ...`. It plays no role in bucket deletion. It's primarily used by the `rgw-orphan-list` tool. I will investigate those logged warning messages further.

Actions
Copy link
 #3

Updated by J. Eric Ivancich over 2 years ago

Are you deleting the bucket via s3 or swift? Or via radosgw-admin?

If s3 or swift, would you raise the logging level of your RGWs to 20 and re-issue the bucket delete?

If radosgw-admin, would you issue the command with the "--debug-rgw=20" command-line option?

Either way, please send the relevant segment of the logs. Thanks!

Actions
Copy link
 #4

Updated by Paul Jurco over 2 years ago

J. Eric Ivancich wrote:

Are you deleting the bucket via s3 or swift? Or via radosgw-admin?

If s3 or swift, would you raise the logging level of your RGWs to 20 and re-issue the bucket delete?

If radosgw-admin, would you issue the command with the "--debug-rgw=20" command-line option?

Either way, please send the relevant segment of the logs. Thanks!

Hi, I used the radosgw-admin command:

radosgw-admin bucket rm --bucket=feed-v2 --purge-objects --bypass-gc --debug-rgw=20

This was a huge bucket (9tib aprox size) and having 4600+ unfinished uploads, with most of the space used by unfinished uploads.
We could not delete it using a simply bucket rm command and we removed all visible and unfinished objects in it with s3cmd. After that we got stuck in this loop when using bucket rm to delete it.
Please find attached the log containing just the first two warnings of "abort_bucket_multiparts WARNING". The log is gzip archived due to size.
Thank you!
Paul

Actions
Copy link
 #5

Updated by Paul Jurco over 2 years ago

HI!
Tested this problem in prod in octopus 15.2.15 and I get the same result, infinite loop.
15.2.15 includes patch https://tracker.ceph.com/issues/51330
Please advise.
Thank you!
Paul

Actions
Copy link
 #6

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Triaged
Actions
Copy link
 #7

Updated by J. Eric Ivancich over 2 years ago

I'm looking at it, Paul, and will report back.

Actions
Copy link
 #8

Updated by J. Eric Ivancich over 2 years ago

J. Eric Ivancich wrote:

I'm looking at it, Paul, and will report back.

I have a fix. Working on making the appropriate PRs.

Actions
Copy link
 #9

Updated by J. Eric Ivancich over 2 years ago

  • Status changed from Triaged to In Progress
Actions
Copy link
 #10

Updated by J. Eric Ivancich over 2 years ago

Here's the branch with the fix: https://github.com/ivancich/ceph-fork/tree/wip-multipart-purge-fix-octopus

Currently testing master and pacific to see if they suffer from the same issue.

Actions
Copy link
 #11

Updated by J. Eric Ivancich over 2 years ago

J. Eric Ivancich wrote:

Here's the branch with the fix: https://github.com/ivancich/ceph-fork/tree/wip-multipart-purge-fix-octopus

Currently testing master and pacific to see if they suffer from the same issue.

It appears this issue is on pacific and octopus but not master.

Actions
Copy link
 #12

Updated by J. Eric Ivancich over 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 43863
Actions
Copy link
 #13

Updated by J. Eric Ivancich over 2 years ago

  • Affected Versions v15.2.15, v16.2.6 added

Updating versions affected.

Actions
Copy link
 #14

Updated by J. Eric Ivancich over 2 years ago

  • Pull request ID changed from 43863 to 43862
Actions
Copy link
 #15

Updated by J. Eric Ivancich over 2 years ago

  • Backport set to octopus
Actions
Copy link
 #16

Updated by J. Eric Ivancich over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #17

Updated by Backport Bot over 2 years ago

Actions
Copy link
 #18

Updated by Loïc Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link
 #19

Updated by Paul Jurco about 2 years ago

J. Eric Ivancich wrote:

Here's the branch with the fix: https://github.com/ivancich/ceph-fork/tree/wip-multipart-purge-fix-octopus

Currently testing master and pacific to see if they suffer from the same issue.

I can confirm it worked on 15.2.16 on the bucket we could not delete:

# radosgw-admin bucket rm --bucket=feed-v2 --purge-objects --bypass-gc
2022-03-08T10:37:37.981+0200 7f15d12e2080  2 all 8 watchers are set, enabling cache
2022-03-08T10:37:37.981+0200 7f15927e4700  2 RGWDataChangesLog::ChangesRenewThread: start
2022-03-08T10:37:41.113+0200 7f15d12e2080  0 abort_bucket_multiparts WARNING : aborted 1 incomplete multipart uploads
2022-03-08T10:37:41.609+0200 7f15d12e2080  0 abort_bucket_multiparts WARNING : aborted 2 incomplete multipart uploads
2022-03-08T10:37:44.757+0200 7f15927e4700  2 RGWDataChangesLog::ChangesRenewThread: start
2022-03-08T10:37:44.781+0200 7f15d12e2080  2 removed watcher, disabling cache

Thank you!

Actions
Copy link

Also available in: Atom PDF