Project

General

Profile

Actions
Copy link

Bug #46242

open

rados -p default.rgw.buckets.data returning over millions objects No such file or directory

Added by Manuel Rios almost 4 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v14.2.9
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi Dev,

Due sharding / s3 bugs we synced the bucket of customer to new ones.

Once we tryed to delete we're unable to do it.

Then we listed all objects using rados -p POOL ls of the s3 pool with grep :

That ID is the marker of the bucket to be deleted.
Previusly to use that option we tryed radosgw-admin bucket rm --bypass-gc --inconsistent-index etc...

rados -p default.rgw.buckets.data ls | grep 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62

This query, return us a 3 millions objects including _shadows and _multiparts from the bucket.

Once finished we piped each string returned like :

48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo

to 
rados -p default.rgw.buckets.data rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo

And we're getting :

error removing default.rgw.buckets.data>48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo: (2) No such file or directory

This happened in the 99% of files.

We think is something related to parse the string at some level, but we're unable to debug it and cleanup.

Actually this case waste near 100TB of objects unable to be deleted.

Regards
Manuel

Actions
Copy link
 #1

Updated by Josh Durgin almost 4 years ago

Can you verify the object name is passed to 'rados rm' correctly by enclosing it in single quotes?

Is it possible rgw garbage collected the object between the time you listed it and tried to remove it?

Actions
Copy link
 #2

Updated by Manuel Rios almost 4 years ago

Hi Josh,

Object still exist:

Just done your test and failed with single quotes -> https://gyazo.com/a04fcf5b52216048016c8c9974e0ed6d

Do you want to check online? I can share my screen over bluejeans.

Regards

Actions
Copy link
 #3

Updated by Vikhyat Umrao almost 4 years ago

I think first you should verify the correct name as Josh suggested with `rados stat` command.

For example I did try to create the object with the same name, then stat and rm and it works.

[root@dell-per630-13 ~]# rados -p us-east.rgw.log put 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo anaconda-ks.cfg

[root@dell-per630-13 ~]# rados -p us-east.rgw.log stat 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo
us-east.rgw.log/48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo mtime 2020-07-20T13:37:35.000000-0400, size 1409

[root@dell-per630-13 ~]# rados -p us-east.rgw.log rm 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo

[root@dell-per630-13 ~]# rados -p us-east.rgw.log stat 48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo
 error stat-ing us-east.rgw.log/48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.62__shadow_MBS-5b1d33ba-e366-4781-a434-5ab42599b1f8/CBB_USUARI-HP/CBB_DiskImage/Disk_00000000-0000-0000-0000-000000000000/Volume_NTFS_00000000-0000-0000-0000-000000000001$/20191113152201/6.cbrevision.kwesRK89yEq0gQ__udNiZDhOiftYXUo: (2) No such file or directory
Actions
Copy link

Also available in: Atom PDF