Project

General

Profile

Bug #42051

possible memory leak while resharding

Added by Mark Kogan 5 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

If resharding is started with the parameters 'verbose=true' and 'out=nullptr', like in the example:

ret = br.execute(entry.new_num_shards, max_entries, true, nullptr, formatter, this);
                                                    ^ verb|ose
                                                          ^out

Then in src/rgw/rgw_reshard.cc , function:

int RGWBucketReshard::do_reshard(...
...
for (auto iter = entries.begin(); iter != entries.end(); ++iter) {
    rgw_cls_bi_entry& entry = *iter;
    if (verbose) {
      formatter->open_object_section("entry");

      encode_json("shard_id", i, formatter);
      encode_json("num_entry", total_entries, formatter);
----> encode_json("entry", entry, formatter); 
    }

The formatter is never flushed if out=nullptr
and the "encode_json("entry", entry, formatter);"
is accumulating 1268 byte per object or roughly 1.2 GB Ram per 1000000 objects while resharding

For example content of entry, entry len is 1268 bytes:

entry":{"name":"myobjects27225067","instance":"","ver":{"pool":6,"epoch":151045},"locator":"","exists":"true","meta":{"category":1,"size":4000,"mtime":"2019-09-22 15:06:53.754838Z","etag":"6c1b4100d743a6937ad6c59244d535b3","owner":"cosbench","owner_display_name":"Test Id","content_type":"application/octet-stream","accounted_size":4000,"user_data":""},"tag":"597bae0b-2c94-4fa5-9d46-a0ee7a8c630a.4121.1205919","flags":0,"pending_map":[],"versioned_epoch":0}}}

Screenshot_20190925_100310.jpg View - valgrind massif (764 KB) Mark Kogan, 09/25/2019 12:58 PM


Related issues

Copied to rgw - Backport #42840: nautilus: possible memory leak while resharding Resolved

History

#1 Updated by Mark Kogan 5 months ago

  • Description updated (diff)

#2 Updated by Mark Kogan 5 months ago

  • Pull request ID set to 30575

#4 Updated by Casey Bodley 5 months ago

note that the backport depends on https://github.com/ceph/ceph/pull/29894

#5 Updated by Casey Bodley 5 months ago

  • Status changed from 12 to 7

#6 Updated by Casey Bodley 3 months ago

  • Status changed from 7 to Pending Backport
  • Backport set to nautilus

#7 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #42840: nautilus: possible memory leak while resharding added

#8 Updated by Nathan Cutler about 1 month ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF