Actions
Bug #49793
openrgw gc object leak when gc omap set entry failed with a large omap value
Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Updated by dovefi Z about 3 years ago
how to reproduce¶
1. upload a large file about 1.3TB,the object name length is about 100 char.
s3cmd put database/173727306/20210115/mysql_newsnapshot_g0_200037_20210115023303.tar.gz s3://large-bucket
2. delete object
s3cmd del s3://large-bucket/database/173727306/20210115/mysql_newsnapshot_g0_200037_20210115023303.tar.gz
3. the gc list is empty
radosgw-admin gc list --include-all []
4. rados df
$ rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR default.rgw.buckets.data 1.3TiB 340787 0 1022361 0 0 0 80887124 58.3GiB 207278635 183TiB
so the object have been leak, and finlly we have found the reaseon,bellow is the osd log
do_op msg data len 128255528 > osd_max_write_size 94371840 on osd_op(client.2308860.0:7839785 60.7b 60:deff95d7:::gc.27:head [call rgw.gc_set_entry] snapc 0=[] ondisk+write+known_if_redirected e15159) v8
the omap is too large for the osd_max_write_size limit, The problem can be solved by increasing the parameter value, But what if the file gets bigger and bigger?may be save object manifest in gc omap value is better。
Updated by Konstantin Shalygin 6 months ago
- Project changed from ceph-volume to rgw
- Affected Versions v18.2.0 added
- Affected Versions deleted (
v10.0.0, v10.1.1, v10.2.0, v10.2.1, v10.2.10, v10.2.11, v10.2.12, v10.2.2, v10.2.3, v10.2.4, v10.2.5, v10.2.6, v10.2.7, v10.2.8, v10.2.9, v11.1.0, v11.2.0, v11.2.1, v11.2.2, v12.0.0, v12.1.0, v12.2.0, v12.2.1, v12.2.10, v12.2.11, v12.2.12, v12.2.13, v12.2.14, v12.2.2, v12.2.3, v12.2.4, v12.2.5, v12.2.6, v12.2.7, v12.2.8, v12.2.9, v13.0.0, v13.2.0, v13.2.1, v13.2.10, v13.2.11, v13.2.2, v13.2.3, v13.2.4, v13.2.5, v13.2.6, v13.2.7, v13.2.8, v13.2.9, v14.0.0, v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.12, v14.2.13, v14.2.14, v14.2.15, v14.2.16, v14.2.17, v14.2.18, v14.2.2, v14.2.3, v14.2.4, v14.2.5, v14.2.6, v14.2.7, v14.2.8, v14.2.9, v15.0.0, v15.2.1, v15.2.10, v15.2.2, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8, v15.2.9, v16.0.0, v16.0.1, v16.1.0, v16.1.1, v17.0.0)
Actions