Feature #138
closedTry out tcmalloc
Description
The cosd daemon seems to eat up fragmented memory or something, since heap size stays fairly consistent but top memory usage keeps growing. johnl in the channel suggested we look at tcmalloc (http://goog-perftools.sourceforge.net/doc/tcmalloc.html) and the description certainly sounds promising.
I'm concerned by a sentence under the caveats section, though: "TCMalloc currently does not return any memory to the system."
But http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html seems to be more up-to-date and omits that one, so maybe we're good.
Anyway, it should be looked into!
Updated by Greg Farnum almost 14 years ago
- Status changed from New to In Progress
Okay, this is definitely something we need to look into more. I tried running with tcmalloc and the standard malloc, and ran
"./rados -p data bench 3600 write -t 20 -b 10485760" (1 hour run, 10MB writes, 20 in-flight)
The standard malloc resulted in the OSD using 3.1GB resident and 3.6GB virtual memory at the end of the hour (peaking at 3.3GB and 3.7GB).
tcmalloc resulted in the OSD using 703.7MB resident and 1.0GB virtual at the end of the hour (that was the peak, but it did clean up at itself several times over the run).
Updated by Greg Farnum almost 14 years ago
- Priority changed from Low to Normal
Going to implement this as a compile-time option (based on available libraries) once I've audited the mds and osd for memory leaks so we can see if ptmalloc holds up better once those are toned down.
Updated by Greg Farnum almost 14 years ago
All right, this is on hold while we work through some of the bugs that have been reported recently.
Updated by Sage Weil over 13 years ago
let's turn this on for cmds and cosd.
and update configure.ac to detect it.
and set debian/control and ceph.spec.in dependencies.
Updated by Greg Farnum over 13 years ago
Dunno how to set up the packaging stuff, but the configure.ac/Makefile stuff wasn't too complicated and is pushed in the tcmalloc branch.
Updated by Wido den Hollander over 13 years ago
I can confirm that tcmalloc() works fine, seeing about 70% memory reduction on my OSD and MDS, great!
Tested it on Ubuntu 10.04 AMD64, but there was a problem with the libgoogle* packages from Ubuntu are not correct: https://bugs.launchpad.net/ubuntu/+source/google-perftools/+bug/359736
Installing libgoogle* libtcmalloc and libunwind from Debian Sid works.
Updated by Greg Farnum over 13 years ago
- Assignee changed from Greg Farnum to Sage Weil
I guess Sage is going to handle the packaging once the kernel rbd split is accomplished.
Updated by Sage Weil over 13 years ago
- Status changed from In Progress to Resolved
merged by a5e582df85354d57b178697b7ad13a75078edb1e