Project

General

Profile

Bug #47241

still unable to delete "empty buckets"

Added by Manuel Rios about 1 year ago. Updated about 1 year ago.

Status:
Triaged
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi,

Im trying to delete "empty" bucket for customer. I know this bucket got several problems due MPU / sharding bugs.

ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)

radosgw-admin bucket rm --bucket=ControlGroup --purge-objects --debug-rgw=20


2020-09-01 20:03:30.402 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.402 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20191014190335/86.cbrevision.2~rDh73BVAlj8Un1_oB3NIFOR2RBlxafQ.meta" for cleaning up
2020-09-01 20:03:30.402 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.402 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~AgB-iqelYF55YURu_Ko-sRrDYjEQ1yK.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~Gdp70Y3tqIDyRM5dPJ_HqBiXqiuqLak.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~JYh5dztulEMctk2PMcxhb1JWPuYwdNo.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~fvgpX4dxT4TFDIF8LRDRN6CT5DL9b6y.meta" for cleaning up

What that error cli start a loop that never finish deleting affected buckets.

Devs: is there any way to create a syntethic .meta to allow process to continue?

History

#1 Updated by Casey Bodley about 1 year ago

Manuel Rios wrote:

Devs: is there any way to create a syntethic .meta to allow process to continue?

It looks like you can use the `rados touch` command to create empty .meta objects. Then abort_multipart_parts() should call list_multipart_parts(), find no parts, and return success.

#2 Updated by Manuel Rios about 1 year ago

Hi Casey,

Do you think that is possible in next release , when you specify --purge-objects also generate syntethic .meta? As autofix delete bucket feature.. should be good.

Because actually i need to run debug-rgw=20, pipe to a log file, grep all unable files and rados touch...

As --purge-objects will be a yes-yes condition that should remove the bucket even if they got multiparts problems due sharding during uploads.

#3 Updated by Manuel Rios about 1 year ago

Should i include bucket marker as string when creating object using rados touch no?

#4 Updated by Manuel Rios about 1 year ago

That solution finally didnt work. I wiped the full cluster. Thanks

#5 Updated by Casey Bodley about 1 year ago

  • Status changed from New to Triaged

the 'bucket rm' needs an audit of its error paths. most error paths should not be fatal, and should continue deleting the objects it can reach

#6 Updated by Or Friedmann about 1 year ago

  • Assignee set to Or Friedmann

Also available in: Atom PDF