Ceph » RADOS

Each osd consumes too much memory when i tested ec overwrite. So i watched heap memory with google-perftools.
I found that four function below allocated almost 60MB memory each of them.

     0.0   0.0% 100.0%     63.5  12.8% ECUtil::HashInfo::encode
     0.0   0.0% 100.0%     57.6  11.6% SnapSet::encode
     0.0   0.0% 100.0%     57.6  11.6% object_info_t::encode
     0.0   0.0% 100.0%     63.5  12.8% ObjectModDesc::setattrs

When i reviewed the code, i found the problem in function ECTransaction::generate_transactions.
A、 we encode HashInfo and put it into a bufferlist named old_hinfo. The size of old_hinfo is 4kB.
and then, insert it to xattr_rollback

  bufferlist old_hinfo;
      ::encode(*hinfo, old_hinfo);
      xattr_rollback[ECUtil::get_hinfo_key()] = old_hinfo;

B、we do the same to snapset and object_info_t.

        if (obc) {
          xattr_rollback.insert(  //attr_cache includes snapset and object_info_t
            obc->attr_cache.begin(),
            obc->attr_cache.end());
          obc->attr_cache.clear();
        }

snapset and object_info_t was encoded at function PrimaryLogPG::finish_ctx. they has been store in bufferlist

independently, the bufferlist size also 4KB

C、Eventually, xattr_rollback is inserted into entry->mod_desc.bl.

 if (entry && !xattr_rollback.empty()) {
        entry->mod_desc.setattrs(xattr_rollback); //xattr_rollback will be encode into entry->mod_desc.bl.
      }

until now, we have allocated four bufferlist. bufferlists cost at least 16KB spaces, It won't be released until the pg entries destroyed.
In fact, it only take less then 1kB spaces to store all of they, too many space is watsed.

I fixed this by trimming entry->mod_desc.bl after attr_cache has been set.
so I add a line at the bottom of function.

      if (!op.is_delete()) {
        bufferlist hbuf;
        ::encode(*hinfo, hbuf);
        for (auto &&i : *transactions) {
          i.second.setattr(
            coll_t(spg_t(pgid, i.first)),
            ghobject_t(oid, ghobject_t::NO_GEN, i.first),
            ECUtil::get_hinfo_key(),
            hbuf);
        }
      }
      entry->mod_desc.trim_bl();

and watch heap memory again.
now is good.

     0.0   0.0% 100.0%      2.6   1.2% SnapSet::encode
     0.0   0.0% 100.0%      2.6   1.2% object_info_t::encode
     0.0   0.0% 100.0%      0.0   0.0% ECUtil::HashInfo::encode
     0.0   0.0% 100.0%      0.1   0.1% ObjectModDesc::setattrs

more then 200MB memory spaces is saved.

Maybe we should fix it?