Project

General

Profile

Actions

Bug #46242

open

rados -p default.rgw.buckets.data returning over millions objects No such file or directory

Added by Manuel Rios almost 4 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi Dev,

Due sharding / s3 bugs we synced the bucket of customer to new ones.

Once we tryed to delete we're unable to do it.

Then we listed all objects using rados -p POOL ls of the s3 pool with grep :

That ID is the marker of the bucket to be deleted.
Previusly to use that option we tryed radosgw-admin bucket rm --bypass-gc --inconsistent-index etc...

rados -p default.rgw.buckets.data ls | grep 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62

This query, return us a 3 millions objects including _shadows and _multiparts from the bucket.

Once finished we piped each string returned like :

48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo

to
rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo

And we're getting :

error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo: (2) No such file or directory

This happened in the 99% of files.

We think is something related to parse the string at some level, but we're unable to debug it and cleanup.

Actually this case waste near 100TB of objects unable to be deleted.

Regards
Manuel

Actions

Also available in: Atom PDF