Bug #42051
Updated by Mark Kogan over 4 years ago
If resharding is started with the parameters 'verbose=true' and 'out=nullptr', like in the example:
<pre>
ret = br.execute(entry.new_num_shards, max_entries, true, nullptr, formatter, this);
^ verb|ose
^out
</pre>
Then in src/rgw/rgw_reshard.cc , function:
<pre><code class="c">
int RGWBucketReshard::do_reshard(...
...
for (auto iter = entries.begin(); iter != entries.end(); ++iter) {
rgw_cls_bi_entry& entry = *iter;
if (verbose) {
formatter->open_object_section("entry");
encode_json("shard_id", i, formatter);
encode_json("num_entry", total_entries, formatter);
----> encode_json("entry", entry, formatter);
}
</code></pre>
The formatter is never flushed if out=nullptr
and the "encode_json("entry", entry, formatter);"
is accumulating 1268 byte per object or roughly 1.2 GB Ram per 1000000 objects while resharding
For example content of entry, entry len is 1268 bytes:
<pre>
entry":{"name":"myobjects27225067","instance":"","ver":{"pool":6,"epoch":151045},"locator":"","exists":"true","meta":{"category":1,"size":4000,"mtime":"2019-09-22 15:06:53.754838Z","etag":"6c1b4100d743a6937ad6c59244d535b3","owner":"cosbench","owner_display_name":"Test Id","content_type":"application/octet-stream","accounted_size":4000,"user_data":""},"tag":"597bae0b-2c94-4fa5-9d46-a0ee7a8c630a.4121.1205919","flags":0,"pending_map":[],"versioned_epoch":0}}}
</pre>