how to reproduce¶
1. upload a large file about 1.3TB,the object name length is about 100 char.
s3cmd put database/173727306/20210115/mysql_newsnapshot_g0_200037_20210115023303.tar.gz s3://large-bucket
2. delete object
s3cmd del s3://large-bucket/database/173727306/20210115/mysql_newsnapshot_g0_200037_20210115023303.tar.gz
3. the gc list is empty
radosgw-admin gc list --include-all
[]
4. rados df
$ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
default.rgw.buckets.data 1.3TiB 340787 0 1022361 0 0 0 80887124 58.3GiB 207278635 183TiB
so the object have been leak, and finlly we have found the reaseon,bellow is the osd log
do_op msg data len 128255528 > osd_max_write_size 94371840 on osd_op(client.2308860.0:7839785 60.7b 60:deff95d7:::gc.27:head [call rgw.gc_set_entry] snapc 0=[] ondisk+write+known_if_redirected e15159) v8
the omap is too large for the osd_max_write_size limit, The problem can be solved by increasing the parameter value, But what if the file gets bigger and bigger?may be save object manifest in gc omap value is better。