Bug #47241
openstill unable to delete "empty buckets"
0%
Description
Hi,
Im trying to delete "empty" bucket for customer. I know this bucket got several problems due MPU / sharding bugs.
ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
radosgw-admin bucket rm --bucket=ControlGroup --purge-objects --debug-rgw=20
2020-09-01 20:03:30.402 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.402 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20191014190335/86.cbrevision.2~rDh73BVAlj8Un1_oB3NIFOR2RBlxafQ.meta" for cleaning up
2020-09-01 20:03:30.402 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.402 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~AgB-iqelYF55YURu_Ko-sRrDYjEQ1yK.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~Gdp70Y3tqIDyRM5dPJ_HqBiXqiuqLak.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~JYh5dztulEMctk2PMcxhb1JWPuYwdNo.meta" for cleaning up
2020-09-01 20:03:30.403 7f9ecf77e840 20 abort_multipart_upload: list_multipart_parts returned -2
2020-09-01 20:03:30.403 7f9ecf77e840 10 abort_bucket_multiparts NOTE : unable to find part(s) of aborted multipart upload of "_multipart_MBS-00a824a2-334a-4cba-91df-167262738728/CBB_ERECAJ/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000000$/20200104200301/179.cbrevision.2~fvgpX4dxT4TFDIF8LRDRN6CT5DL9b6y.meta" for cleaning up
What that error cli start a loop that never finish deleting affected buckets.
Devs: is there any way to create a syntethic .meta to allow process to continue?
Updated by Casey Bodley over 3 years ago
Manuel Rios wrote:
Devs: is there any way to create a syntethic .meta to allow process to continue?
It looks like you can use the `rados touch` command to create empty .meta objects. Then abort_multipart_parts() should call list_multipart_parts(), find no parts, and return success.
Updated by Manuel Rios over 3 years ago
Hi Casey,
Do you think that is possible in next release , when you specify --purge-objects also generate syntethic .meta? As autofix delete bucket feature.. should be good.
Because actually i need to run debug-rgw=20, pipe to a log file, grep all unable files and rados touch...
As --purge-objects will be a yes-yes condition that should remove the bucket even if they got multiparts problems due sharding during uploads.
Updated by Manuel Rios over 3 years ago
Should i include bucket marker as string when creating object using rados touch no?
Updated by Manuel Rios over 3 years ago
That solution finally didnt work. I wiped the full cluster. Thanks
Updated by Casey Bodley over 3 years ago
- Status changed from New to Triaged
the 'bucket rm' needs an audit of its error paths. most error paths should not be fatal, and should continue deleting the objects it can reach