Project

General

Profile

Bug #38470

Radosg-admin can't delete bucket and radosgw-admin gc process not working (or very slowly)

Added by hoan nv about 5 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi all.

I have a ceph cluster. After delete a bucket has 100M object, radosgw-admin gc list has 1419659 objects.

I run radosgw-admin gc process, it not reduce number objects.

radosgw-admin bucket rm --bucket=sample_bucket --purge-objects --debug-ms=1

2019-02-25 15:14:00.688 7fd8e7c02700  1 -- 172.24.9.51:0/3200411148 <== osd.125 172.24.8.54:6826/3486989 2 ==== osd_op_reply(108 17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_tienhv.jpg [stat,cmpxattr (32) op 1 mode 1,setxattr (14)] v25102'85540 uv85540 ondisk = 0) v8 ==== 287+0+0 (2785563410 0 0) 0x7fd8dc00f320 con 0x2b85560
2019-02-25 15:14:00.688 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:109 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.guard_bucket_resharding,call rgw.bucket_unlink_instance] snapc 0=[] ondisk+write+known_if_redirected e25102) v8 -- 0x2b8efb0 con 0
2019-02-25 15:14:00.688 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 16 ==== osd_op_reply(109 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call,call] v25102'4663291 uv4662993 ondisk = 0) v8 ==== 241+0+0 (2028355245 0 0) 0x7fd8e0014c10 con 0x7fd8e000c080
2019-02-25 15:14:00.688 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:110 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.guard_bucket_resharding,call rgw.bucket_read_olh_log] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b87bc0 con 0
2019-02-25 15:14:00.688 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 17 ==== osd_op_reply(110 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call,call] v0'0 uv4662993 ondisk = 0) v8 ==== 241+0+11 (1961857954 0 1993775135) 0x7fd8e00049e0 con 0x7fd8e000c080
2019-02-25 15:14:00.688 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:111 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.bi_get] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b7af10 con 0
2019-02-25 15:14:00.692 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 18 ==== osd_op_reply(111 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call] v0'0 uv4662993 ondisk = 0) v8 ==== 199+0+148 (1347967863 0 1998127041) 0x7fd8e00049e0 con 0x7fd8e000c080
2019-02-25 15:14:00.692 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.53:6853/2229354 -- osd_op(unknown.0.0:112 18.9aa 18:5597dd5f:::17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_vtv.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b7af10 con 0
2019-02-25 15:14:01.984 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.85 172.24.8.53:6853/2229354 1 ==== osd_op_reply(112 17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_vtv.jpg [getxattrs,stat] v0'0 uv271611 ondisk = 0) v8 ==== 242+0+13659717 (2678204839 0 740394784) 0x7fd8e0014c10 con 0x2b87bc0
2019-02-25 15:14:02.300 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.53:6853/2229354 -- osd_op(unknown.0.0:113 18.9aa 18:5597dd5f:::17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_vtv.jpg:head [stat,cmpxattr user.rgw.olh.idtag (32) op 1 mode 1,setxattr user.rgw.olh.pending.000000005c73a3cai8mixbj151q569dn (14)] snapc 0=[] ondisk+write+known_if_redirected e25102) v8 -- 0x5ace970 con 0
2019-02-25 15:14:02.580 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.85 172.24.8.53:6853/2229354 2 ==== osd_op_reply(113 17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_vtv.jpg [stat,cmpxattr (32) op 1 mode 1,setxattr (14)] v25102'271612 uv271612 ondisk = 0) v8 ==== 284+0+0 (1268605200 0 0) 0x7fd8e0013cb0 con 0x2b87bc0
2019-02-25 15:14:02.580 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:114 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.guard_bucket_resharding,call rgw.bucket_unlink_instance] snapc 0=[] ondisk+write+known_if_redirected e25102) v8 -- 0x5ace600 con 0
2019-02-25 15:14:02.584 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 19 ==== osd_op_reply(114 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call,call] v25102'4663292 uv4662993 ondisk = 0) v8 ==== 241+0+0 (710074718 0 0) 0x7fd8e0013cb0 con 0x7fd8e000c080
2019-02-25 15:14:02.584 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:115 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.guard_bucket_resharding,call rgw.bucket_read_olh_log] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x5acdf70 con 0
2019-02-25 15:14:02.584 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 20 ==== osd_op_reply(115 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call,call] v0'0 uv4662993 ondisk = 0) v8 ==== 241+0+11 (1576923740 0 1993775135) 0x7fd8e0013cb0 con 0x7fd8e000c080
2019-02-25 15:14:02.652 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:116 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b7af10 con 0
2019-02-25 15:14:02.668 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 21 ==== osd_op_reply(116 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call] v0'0 uv4662993 ondisk = 0) v8 ==== 199+0+446 (1331265696 0 4111972205) 0x7fd8e0013cb0 con 0x7fd8e000c080
2019-02-25 15:14:02.668 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.21:6801/1844268 -- osd_op(unknown.0.0:117 16.61 16:86e31656:::.dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0:head [call rgw.bi_get] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b7af10 con 0
2019-02-25 15:14:02.672 7fd8e8c04700  1 -- 172.24.9.51:0/3200411148 <== osd.29 172.24.8.21:6801/1844268 22 ==== osd_op_reply(117 .dir.17a4ce99-009e-40f2-a2d2-2afc218ebd9b.216060181.1.0 [call] v0'0 uv4662993 ondisk = 0) v8 ==== 199+0+148 (1347967863 0 1998127041) 0x7fd8e0013cb0 con 0x7fd8e000c080
2019-02-25 15:14:02.672 7fd8fa50d700  1 -- 172.24.9.51:0/3200411148 --> 172.24.8.53:6853/2229354 -- osd_op(unknown.0.0:118 18.9aa 18:5597dd5f:::17a4ce99-009e-40f2-a2d2-2afc218ebd9b.1427945.187_vtv.jpg:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x2b7af10 con 0

radosgw-admin gc process --include-all --debug-ms=1

1) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.128 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1149 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20aa8d0 con 0
2019-02-25 15:14:48.128 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1042 ==== osd_op_reply(1149 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.128 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1150 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20ab2e0 con 0
2019-02-25 15:14:48.128 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1043 ==== osd_op_reply(1150 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.128 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1151 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20aa8d0 con 0
2019-02-25 15:14:48.132 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1044 ==== osd_op_reply(1151 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.132 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1152 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20ab2e0 con 0
2019-02-25 15:14:48.132 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1045 ==== osd_op_reply(1152 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.132 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1153 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20aa8d0 con 0
2019-02-25 15:14:48.132 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1046 ==== osd_op_reply(1153 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.132 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1154 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20ab2e0 con 0
2019-02-25 15:14:48.136 7f16a7a32700  1 -- 172.24.9.51:0/438005962 <== osd.4 172.24.8.21:6806/1858502 1047 ==== osd_op_reply(1154 gc.12 [call] v0'0 uv1093746 ondisk = 0) v8 ==== 149+0+6138 (3003091281 0 101532591) 0x7f16a002a840 con 0x1ff0fb0
2019-02-25 15:14:48.136 7f16b9b3c700  1 -- 172.24.9.51:0/438005962 --> 172.24.8.21:6806/1858502 -- osd_op(unknown.0.0:1155 15.d5 15:ab569f81:gc::gc.12:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e25102) v8 -- 0x20aa8d0 con 0

I think 2 problem is ralated.

Thanks.


Related issues

Related to rgw - Bug #38134: rgw: `radosgw-admin bucket rm ... --purge-objects` can hang... Resolved 01/31/2019
Related to rgw - Bug #38454: rgw: gc entries with zero-length chains are not cleaned up Resolved 02/22/2019

History

#1 Updated by Abhishek Lekshmanan about 5 years ago

Can you post teh radosgw-admin gc list --include-all output. Also note another fix in this area https://tracker.ceph.com/issues/38454

#2 Updated by Casey Bodley about 5 years ago

  • Related to Bug #38134: rgw: `radosgw-admin bucket rm ... --purge-objects` can hang... added

#3 Updated by Casey Bodley about 5 years ago

The bucket rm issue looks like the one we fixed in http://tracker.ceph.com/issues/38134

#4 Updated by hoan nv about 5 years ago

radosgw-admin gc list --include-all out is nothing. It just return 0.
no gc log deleted.

#5 Updated by Casey Bodley about 5 years ago

  • Related to Bug #38454: rgw: gc entries with zero-length chains are not cleaned up added

#6 Updated by Casey Bodley about 5 years ago

  • Status changed from New to 12
  • Assignee set to J. Eric Ivancich

it looks like http://tracker.ceph.com/issues/38454 was causing infinite loops in gc process, so i think that's what you're seeing here

#7 Updated by J. Eric Ivancich about 5 years ago

@hoan nv:

Once the backport of http://tracker.ceph.com/issues/38713 is complete, would you kindly test it? This seems likely to be a duplicate.

#8 Updated by hoan nv about 5 years ago

Eric Ivancich wrote:

@hoan nv:

Once the backport of http://tracker.ceph.com/issues/38713 is complete, would you kindly test it? This seems likely to be a duplicate.

Yes.I am planning to update ceph.

Thanks.

#9 Updated by hoan nv almost 5 years ago

i updated my cluster to 13.2.5

issue https://tracker.ceph.com/issues/38134 not same my issue

delete bucket log :

2019-03-31 22:34:36.797 7fd2164d0300 10 RGWRados::cls_bucket_list_ordered: got _multipart_multipart.mp4.2~yFVvAfSuyxKFdpkSXI2fTBKJJM5ip_2.1[]
2019-03-31 22:34:36.797 7fd2164d0300 10 RGWRados::cls_bucket_list_ordered: got _multipart_multipart.mp4.2~yFVvAfSuyxKFdpkSXI2fTBKJJM5ip_2.meta[]
2019-03-31 22:34:36.817 7fd2164d0300  0 WARNING : aborted 17 incomplete multipart uploads
2019-03-31 22:34:36.817 7fd2164d0300 -1 ERROR: unable to remove bucket(2009) Unknown error 2009
2019-03-31 22:34:36.817 7fd2164d0300 20 remove_watcher() i=0
2019-03-31 22:34:36.817 7fd2164d0300  2 removed watcher, disabling cache

#10 Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New

#11 Updated by jack jack about 4 years ago

HI , I have same issues too
Version : ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)

I delete large object from my bucket that i get warning

LARGE_OMAP_OBJECTS 1 large omap objects
    1 large objects found in pool 'default.rgw.buckets.index'

I check my gc list is null and my bucket is null too when delete complete .

but when i delete my bucket i get ERROR: unable to remove bucket(2009) Unknown error 2009 error i think same to your issue but i don't know how to fix it.

delete bucket log

radosgw-admin bucket rm --bucket=sample_bucket --purge-objects --debug-ms=1

2020-01-17 14:45:56.109 7f11035e1700  1 -- 10.0.222.29:0/769924328 <== osd.48 10.0.201.58:6804/1781221 46 ==== osd_op_reply(1381 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.421 [call] v0'0 uv1496846 ondisk = 0) v8 ==== 196+0+110 (624421284 0 1888912861) 0x7f10fc0273d0 con 0x55580e0dd860
2020-01-17 14:45:56.109 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.56 10.0.201.58:6820/689982 84 ==== osd_op_reply(1383 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.423 [call] v0'0 uv1752127 ondisk = 0) v8 ==== 196+0+110 (3956127056 0 287700792) 0x7f10f4044260 con 0x7f10f402bd40
2020-01-17 14:45:56.109 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6825/3397321 -- osd_op(unknown.0.0:1385 7.5e 7:7ae0fadd:::.dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.425:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.109 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.58:6825/1260572 -- osd_op(unknown.0.0:1386 7.b 7:d03876f6:::.dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.426:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.109 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.35 10.0.201.50:6812/366083 28 ==== osd_op_reply(1384 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.424 [call] v0'0 uv1481217 ondisk = 0) v8 ==== 196+0+110 (753381198 0 721790246) 0x7f10f40415f0 con 0x7f10f8015980
2020-01-17 14:45:56.109 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.58:6812/3187883 -- osd_op(unknown.0.0:1387 7.3f 7:fc52a468:::.dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.427:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.109 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.57 10.0.201.57:6825/3397321 30 ==== osd_op_reply(1385 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.425 [call] v0'0 uv1070695 ondisk = 0) v8 ==== 196+0+110 (2023261328 0 2997045324) 0x7f10f803d5d0 con 0x55580e0d8fa0
2020-01-17 14:45:56.109 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.58:6820/689982 -- osd_op(unknown.0.0:1388 7.3 7:c18107ac:::.dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.428:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.109 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.58 10.0.201.58:6825/1260572 50 ==== osd_op_reply(1386 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.426 [call] v0'0 uv1595079 ondisk = 0) v8 ==== 196+0+110 (636410273 0 1065437772) 0x7f10f802f430 con 0x7f10f4013850
2020-01-17 14:45:56.109 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6800/3109382 -- osd_op(unknown.0.0:1389 7.12 7:4835f492:::.dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.429:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.109 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.52 10.0.201.58:6812/3187883 45 ==== osd_op_reply(1387 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.427 [call] v0'0 uv1519592 ondisk = 0) v8 ==== 196+0+110 (1694743069 0 3852380010) 0x7f10f4044260 con 0x55580e1dfd20
2020-01-17 14:45:56.109 7f11035e1700  1 -- 10.0.222.29:0/769924328 <== osd.45 10.0.201.57:6800/3109382 24 ==== osd_op_reply(1389 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.429 [call] v0'0 uv1912127 ondisk = 0) v8 ==== 196+0+110 (4109050811 0 1536062217) 0x7f10fc032ca0 con 0x55580e222410
2020-01-17 14:45:56.109 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.56 10.0.201.58:6820/689982 85 ==== osd_op_reply(1388 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.428 [call] v0'0 uv1763385 ondisk = 0) v8 ==== 196+0+110 (512564487 0 1920086855) 0x7f10f40415f0 con 0x7f10f402bd40
2020-01-17 14:45:56.113 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.29 10.0.201.100:6807/631298 84 ==== osd_op_reply(1300 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.340 [call] v0'0 uv1998266 ondisk = 0) v8 ==== 196+0+110 (2951426136 0 515681007) 0x7f10f803d5d0 con 0x55580e1e6bf0
2020-01-17 14:45:56.113 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.29 10.0.201.100:6807/631298 85 ==== osd_op_reply(1351 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.391 [call] v0'0 uv1998278 ondisk = 0) v8 ==== 196+0+110 (3405126227 0 1633780788) 0x7f10f802f430 con 0x55580e1e6bf0
2020-01-17 14:45:56.113 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.29 10.0.201.100:6807/631298 86 ==== osd_op_reply(1356 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.396 [call] v0'0 uv1198142 ondisk = 0) v8 ==== 196+0+110 (1504629581 0 3937684959) 0x7f10f803d5d0 con 0x55580e1e6bf0
2020-01-17 14:45:56.113 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.29 10.0.201.100:6807/631298 87 ==== osd_op_reply(1382 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.422 [call] v0'0 uv1998268 ondisk = 0) v8 ==== 196+0+110 (2175816638 0 1100430540) 0x7f10f802f430 con 0x55580e1e6bf0
2020-01-17 14:45:56.117 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.11 10.0.222.30:6813/3237534 51 ==== osd_op_reply(1350 .dir.2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.7092.1.390 [call] v0'0 uv1679480 ondisk = 0) v8 ==== 196+0+110 (1698450074 0 3771474804) 0x7f10f4044260 con 0x55580e216ba0
2020-01-17 14:45:56.117 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6804/791273 -- osd_op(unknown.0.0:1390 9.13 9:cfc90b53:::2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.4176.1__multipart_uploads%2fwebmove.tar.2~Rsm6kLRrov2Mzyszp5PyxUpYz6ySV-z.meta:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.117 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.47 10.0.201.57:6804/791273 85 ==== osd_op_reply(1390 2a5caa29-64c4-4bc2-90d4-eaad8b49baeb.4176.1__multipart_uploads/webmove.tar.2~Rsm6kLRrov2Mzyszp5PyxUpYz6ySV-z.meta [omap-get-vals] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 257+0+0 (4125934789 0 0) 0x7f10f802f430 con 0x7f10f80255c0
2020-01-17 14:45:56.117 7f1115433740 -1 ERROR: unable to remove bucket(2009) Unknown error 2009
2020-01-17 14:45:56.117 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6820/2978280 -- osd_op(unknown.0.0:1391 2.1f 2:f95f44c2:::notify.0:head [watch unwatch cookie 93836681325024] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1f96e0 con 0
2020-01-17 14:45:56.121 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.55 10.0.201.57:6820/2978280 21 ==== osd_op_reply(1391 notify.0 [watch unwatch cookie 93836681325024] v23811'524 uv522 ondisk = 0) v8 ==== 152+0+0 (1417340415 0 0) 0x7f10f802f430 con 0x7f10f4022700
2020-01-17 14:45:56.121 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.222.30:6801/3244407 -- osd_op(unknown.0.0:1392 2.0 2:05bf5b68:::notify.1:head [watch unwatch cookie 93836681344208] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1f43f0 con 0
2020-01-17 14:45:56.121 7f11035e1700  1 -- 10.0.222.29:0/769924328 <== osd.8 10.0.222.30:6801/3244407 17 ==== osd_op_reply(1392 notify.1 [watch unwatch cookie 93836681344208] v23811'514 uv512 ondisk = 0) v8 ==== 152+0+0 (1825684014 0 0) 0x7f10fc032ca0 con 0x55580e0ebe60
2020-01-17 14:45:56.121 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.100:6805/3455858 -- osd_op(unknown.0.0:1393 2.15 2:a93a5511:::notify.2:head [watch unwatch cookie 93836681362432] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1bade0 con 0
2020-01-17 14:45:56.125 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.25 10.0.201.100:6805/3455858 3 ==== osd_op_reply(1393 notify.2 [watch unwatch cookie 93836681362432] v23811'678 uv676 ondisk = 0) v8 ==== 152+0+0 (148015776 0 0) 0x7f10f4044210 con 0x55580e0f0590
2020-01-17 14:45:56.125 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.222.29:6809/2635962 -- osd_op(unknown.0.0:1394 2.e 2:7759931f:::notify.3:head [watch unwatch cookie 93836681363984] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.125 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.2 10.0.222.29:6809/2635962 4 ==== osd_op_reply(1394 notify.3 [watch unwatch cookie 93836681363984] v23811'377 uv375 ondisk = 0) v8 ==== 152+0+0 (1327505356 0 0) 0x7f10f4044210 con 0x7f10fc008580
2020-01-17 14:45:56.125 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.50:6828/715959 -- osd_op(unknown.0.0:1395 2.d 2:b4812045:::notify.4:head [watch unwatch cookie 93836681382960] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1f96e0 con 0
2020-01-17 14:45:56.129 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.39 10.0.201.50:6828/715959 18 ==== osd_op_reply(1395 notify.4 [watch unwatch cookie 93836681382960] v23811'451 uv449 ondisk = 0) v8 ==== 152+0+0 (2876136321 0 0) 0x7f10f802f430 con 0x55580e0f55c0
2020-01-17 14:45:56.129 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6804/791273 -- osd_op(unknown.0.0:1396 2.3 2:c609908c:::notify.5:head [watch unwatch cookie 93836681383760] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1f43f0 con 0
2020-01-17 14:45:56.129 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.47 10.0.201.57:6804/791273 86 ==== osd_op_reply(1396 notify.5 [watch unwatch cookie 93836681383760] v23811'406 uv404 ondisk = 0) v8 ==== 152+0+0 (2744785082 0 0) 0x7f10f802f430 con 0x7f10f80255c0
2020-01-17 14:45:56.129 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.57:6812/1686807 -- osd_op(unknown.0.0:1397 2.14 2:2b04a3e9:::notify.6:head [watch unwatch cookie 93836681386336] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1bade0 con 0
2020-01-17 14:45:56.133 7f11025df700  1 -- 10.0.222.29:0/769924328 <== osd.51 10.0.201.57:6812/1686807 46 ==== osd_op_reply(1397 notify.6 [watch unwatch cookie 93836681386336] v23811'346 uv344 ondisk = 0) v8 ==== 152+0+0 (3127474860 0 0) 0x7f10f802f430 con 0x7f10f400e050
2020-01-17 14:45:56.133 7f1115433740  1 -- 10.0.222.29:0/769924328 --> 10.0.201.58:6800/992588 -- osd_op(unknown.0.0:1398 2.9 2:93e5b521:::notify.7:head [watch unwatch cookie 93836681390560] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x55580e1b5d90 con 0
2020-01-17 14:45:56.133 7f1102de0700  1 -- 10.0.222.29:0/769924328 <== osd.46 10.0.201.58:6800/992588 18 ==== osd_op_reply(1398 notify.7 [watch unwatch cookie 93836681390560] v23811'378 uv376 ondisk = 0) v8 ==== 152+0+0 (1244864401 0 0) 0x7f10f4044210 con 0x55580e0d41a0

radosgw-admin gc process --include-all --debug-ms=1

2020-01-17 14:50:11.093 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.42 10.0.201.50:6840/1119184 4 ==== osd_op_reply(212 gc.24 [call] v23811'3046115 uv3046114 ondisk = 0) v8 ==== 149+0+0 (1741987207 0 0) 0x7f9578030840 con 0x562db00957f0
2020-01-17 14:50:11.093 7f958d7c9700  1 -- 10.0.222.29:0/600890030 <== osd.19 10.0.201.52:6813/952350 4 ==== osd_op_reply(211 gc.23 [call] v23811'6070957 uv6070953 ondisk = 0) v8 ==== 149+0+0 (3679430775 0 0) 0x7f9580032ba0 con 0x562db0091680
2020-01-17 14:50:11.093 7f958d7c9700  1 -- 10.0.222.29:0/600890030 <== osd.22 10.0.201.52:6825/953952 9 ==== osd_op_reply(214 gc.26 [call] v23811'4965586 uv4965585 ondisk = 0) v8 ==== 149+0+0 (1178170500 0 0) 0x7f9580032ba0 con 0x7f958001c550
2020-01-17 14:50:11.093 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.55 10.0.201.57:6820/2978280 31 ==== osd_op_reply(217 gc.29 [call] v23811'3871308 uv3871305 ondisk = 0) v8 ==== 149+0+0 (4112044686 0 0) 0x7f9578030840 con 0x7f95800218d0
2020-01-17 14:50:11.093 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.58 10.0.201.58:6825/1260572 10 ==== osd_op_reply(216 gc.28 [call] v23811'4734418 uv4734416 ondisk = 0) v8 ==== 149+0+0 (1298746182 0 0) 0x7f9578030840 con 0x7f95800116c0
2020-01-17 14:50:11.093 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.55 10.0.201.57:6820/2978280 32 ==== osd_op_reply(204 gc.16 [call] v23811'5259574 uv5259570 ondisk = 0) v8 ==== 149+0+0 (1524478015 0 0) 0x7f9578030840 con 0x7f95800218d0
2020-01-17 14:50:11.093 7f958dfca700  1 -- 10.0.222.29:0/600890030 <== osd.36 10.0.201.50:6816/366655 16 ==== osd_op_reply(218 gc.30 [call] v23811'5419423 uv5419421 ondisk = 0) v8 ==== 149+0+0 (2084277525 0 0) 0x7f9588025650 con 0x562db007d4c0
2020-01-17 14:50:11.093 7f958dfca700  1 -- 10.0.222.29:0/600890030 <== osd.53 10.0.201.57:6816/2577459 5 ==== osd_op_reply(219 gc.31 [call] v23811'6633607 uv6633598 ondisk = 0) v8 ==== 149+0+0 (2434191138 0 0) 0x7f9588025650 con 0x7f957800d560
2020-01-17 14:50:11.093 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.57:6820/2978280 -- osd_op(unknown.0.0:220 2.1f 2:f95f44c2:::notify.0:head [watch unwatch cookie 94754226100784] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00d4680 con 0
2020-01-17 14:50:11.097 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.55 10.0.201.57:6820/2978280 33 ==== osd_op_reply(220 notify.0 [watch unwatch cookie 94754226100784] v23811'527 uv525 ondisk = 0) v8 ==== 152+0+0 (4195300374 0 0) 0x7f95780349c0 con 0x7f95800218d0
2020-01-17 14:50:11.097 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.222.30:6801/3244407 -- osd_op(unknown.0.0:221 2.0 2:05bf5b68:::notify.1:head [watch unwatch cookie 94754226119968] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00a60b0 con 0
2020-01-17 14:50:11.097 7f958dfca700  1 -- 10.0.222.29:0/600890030 <== osd.8 10.0.222.30:6801/3244407 3 ==== osd_op_reply(221 notify.1 [watch unwatch cookie 94754226119968] v23811'517 uv515 ondisk = 0) v8 ==== 152+0+0 (4282826187 0 0) 0x7f9588029260 con 0x562daffceeb0
2020-01-17 14:50:11.097 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.100:6805/3455858 -- osd_op(unknown.0.0:222 2.15 2:a93a5511:::notify.2:head [watch unwatch cookie 94754226138192] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00d4680 con 0
2020-01-17 14:50:11.101 7f958d7c9700  1 -- 10.0.222.29:0/600890030 <== osd.25 10.0.201.100:6805/3455858 7 ==== osd_op_reply(222 notify.2 [watch unwatch cookie 94754226138192] v23811'681 uv679 ondisk = 0) v8 ==== 152+0+0 (629381020 0 0) 0x7f9580036c80 con 0x562daffd35e0
2020-01-17 14:50:11.101 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.222.29:6809/2635962 -- osd_op(unknown.0.0:223 2.e 2:7759931f:::notify.3:head [watch unwatch cookie 94754226139744] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00a60b0 con 0
2020-01-17 14:50:11.101 7f958d7c9700  1 -- 10.0.222.29:0/600890030 <== osd.2 10.0.222.29:6809/2635962 4 ==== osd_op_reply(223 notify.3 [watch unwatch cookie 94754226139744] v23811'380 uv378 ondisk = 0) v8 ==== 152+0+0 (2752611889 0 0) 0x7f9580036c80 con 0x7f9588008580
2020-01-17 14:50:11.101 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.50:6828/715959 -- osd_op(unknown.0.0:224 2.d 2:b4812045:::notify.4:head [watch unwatch cookie 94754226158720] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00d4680 con 0
2020-01-17 14:50:11.105 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.39 10.0.201.50:6828/715959 3 ==== osd_op_reply(224 notify.4 [watch unwatch cookie 94754226158720] v23811'454 uv452 ondisk = 0) v8 ==== 152+0+0 (3633446745 0 0) 0x7f9578030840 con 0x562daffd8610
2020-01-17 14:50:11.105 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.57:6804/791273 -- osd_op(unknown.0.0:225 2.3 2:c609908c:::notify.5:head [watch unwatch cookie 94754226159520] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00a60b0 con 0
2020-01-17 14:50:11.105 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.47 10.0.201.57:6804/791273 4 ==== osd_op_reply(225 notify.5 [watch unwatch cookie 94754226159520] v23811'409 uv407 ondisk = 0) v8 ==== 152+0+0 (4010711185 0 0) 0x7f9578034d50 con 0x7f95780265c0
2020-01-17 14:50:11.105 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.57:6812/1686807 -- osd_op(unknown.0.0:226 2.14 2:2b04a3e9:::notify.6:head [watch unwatch cookie 94754226162096] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00d4680 con 0
2020-01-17 14:50:11.109 7f9587fff700  1 -- 10.0.222.29:0/600890030 <== osd.51 10.0.201.57:6812/1686807 4 ==== osd_op_reply(226 notify.6 [watch unwatch cookie 94754226162096] v23811'349 uv347 ondisk = 0) v8 ==== 152+0+0 (1633324882 0 0) 0x7f9578030840 con 0x7f958000d050
2020-01-17 14:50:11.109 7f959fe1c740  1 -- 10.0.222.29:0/600890030 --> 10.0.201.58:6800/992588 -- osd_op(unknown.0.0:227 2.9 2:93e5b521:::notify.7:head [watch unwatch cookie 94754226166320] snapc 0=[] ondisk+write+known_if_redirected e23811) v8 -- 0x562db00a60b0 con 0
2020-01-17 14:50:11.109 7f958d7c9700  1 -- 10.0.222.29:0/600890030 <== osd.46 10.0.201.58:6800/992588 18 ==== osd_op_reply(227 notify.7 [watch unwatch cookie 94754226166320] v23811'381 uv379 ondisk = 0) v8 ==== 152+0+0 (866670553 0 0) 0x7f9580032d20 con 0x562daffb7120

If you konw any about this issue please tell me

Thankyou!

#12 Updated by Leo Q over 3 years ago

I'm using Octupus 15.2.3 and experiencing the same issue, the delete log is all like these :

2020-09-17T04:50:36.504+0000 7f2ffbaa31c0  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-09-17T04:50:38.200+0000 7f2ffbaa31c0  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-09-17T04:50:39.556+0000 7f2ffbaa31c0  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
  • my cluster operates good, the bucket I wanted to delete is also working, I can put objects in the bucket.
  • I cannot operate the existing objects, I cannot delete them, boto3 would say objects are deleted but they're not.
  • I was using this bucket to do some heavy work, like putting a 1Gb size file in the bucket every minute, I assume there're tons of file chunks in the bucket.
  • I waited at least a month but the storage size that the bucket is consuming is not shrinking.

Hope this could be solved in the next version or so.

Also available in: Atom PDF