Bug #19815
openRollback/EC log entries take gratuitous amounts of memory
0%
Description
Each osd consumes too much memory when i tested ec overwrite. So i watched heap memory with google-perftools.
I found that four function below allocated almost 60MB memory each of them.
0.0 0.0% 100.0% 63.5 12.8% ECUtil::HashInfo::encode
0.0 0.0% 100.0% 57.6 11.6% SnapSet::encode
0.0 0.0% 100.0% 57.6 11.6% object_info_t::encode
0.0 0.0% 100.0% 63.5 12.8% ObjectModDesc::setattrs
When i reviewed the code, i found the problem in function ECTransaction::generate_transactions.
A、 we encode HashInfo and put it into a bufferlist named old_hinfo. The size of old_hinfo is 4kB.
and then, insert it to xattr_rollback
bufferlist old_hinfo;
::encode(*hinfo, old_hinfo);
xattr_rollback[ECUtil::get_hinfo_key()] = old_hinfo;
B、we do the same to snapset and object_info_t.
if (obc) {
xattr_rollback.insert( //attr_cache includes snapset and object_info_t
obc->attr_cache.begin(),
obc->attr_cache.end());
obc->attr_cache.clear();
}
snapset and object_info_t was encoded at function PrimaryLogPG::finish_ctx. they has been store in bufferlist
independently, the bufferlist size also 4KB
C、Eventually, xattr_rollback is inserted into entry->mod_desc.bl.
if (entry && !xattr_rollback.empty()) {
entry->mod_desc.setattrs(xattr_rollback); //xattr_rollback will be encode into entry->mod_desc.bl.
}
until now, we have allocated four bufferlist. bufferlists cost at least 16KB spaces, It won't be released until the pg entries destroyed.
In fact, it only take less then 1kB spaces to store all of they, too many space is watsed.
I fixed this by trimming entry->mod_desc.bl after attr_cache has been set.
so I add a line at the bottom of function.
if (!op.is_delete()) {
bufferlist hbuf;
::encode(*hinfo, hbuf);
for (auto &&i : *transactions) {
i.second.setattr(
coll_t(spg_t(pgid, i.first)),
ghobject_t(oid, ghobject_t::NO_GEN, i.first),
ECUtil::get_hinfo_key(),
hbuf);
}
}
entry->mod_desc.trim_bl();
and watch heap memory again.
now is good.
0.0 0.0% 100.0% 2.6 1.2% SnapSet::encode
0.0 0.0% 100.0% 2.6 1.2% object_info_t::encode
0.0 0.0% 100.0% 0.0 0.0% ECUtil::HashInfo::encode
0.0 0.0% 100.0% 0.1 0.1% ObjectModDesc::setattrs
more then 200MB memory spaces is saved.
Maybe we should fix it?
Updated by Greg Farnum almost 7 years ago
- Project changed from Ceph to RADOS
- Category set to Performance/Resource Usage
- Component(RADOS) OSD added
Updated by Greg Farnum almost 7 years ago
- Subject changed from A problem about ec module to Rollback/EC log entries take gratuitous amounts of memoru
Updated by Nathan Cutler almost 7 years ago
- Subject changed from Rollback/EC log entries take gratuitous amounts of memoru to Rollback/EC log entries take gratuitous amounts of memory
Updated by hongpeng lu almost 7 years ago
It seemed that this bug has been fixed at version 12.1.0.
https://github.com/ceph/ceph/commit/9da684316630ac1c087e03ca6ec039bd4222c0bd