Ceph

Stefan, could you please try (for your monitor's IP and PORT):

ceph -m IP:PORT heap stats

ceph -m IP:PORT heap release

ceph -m IP:PORT heap stats

and see if there are any significant changes?

[: ~]# pmap -x 11783|tail -n1
total kB 1547412 688752 685152

[: ~]# ceph m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 15658448 ( 14.9 MB) Bytes in use by application
MALLOC: 33247232 ( 31.7 MB) Bytes free in page heap
MALLOC: 17707008 ( 16.9 MB) Bytes unmapped in page heap
MALLOC: 2514032 ( 2.4 MB) Bytes free in central cache
MALLOC: 157184 ( 0.1 MB) Bytes free in transfer cache
MALLOC: 14671808 ( 14.0 MB) Bytes free in thread caches
MALLOC: 4675 Spans in use
MALLOC: 305 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------

[: ~]# ceph -m 10.255.0.100:6789 heap release
mon.a releasing free RAM back to system.

[: ~]# ceph m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 16484672 ( 15.7 MB) Bytes in use by application
MALLOC: 4272128 ( 4.1 MB) Bytes free in page heap
MALLOC: 45830144 ( 43.7 MB) Bytes unmapped in page heap
MALLOC: 2543512 ( 2.4 MB) Bytes free in central cache
MALLOC: 161792 ( 0.2 MB) Bytes free in transfer cache
MALLOC: 14663464 ( 14.0 MB) Bytes free in thread caches
MALLOC: 4675 Spans in use
MALLOC: 305 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------

So actual usage (i restartet yesterday) is around 700MB. It's still raising every hour (around 30-50MB).

not it's using 1GB should i run these commands again?

not should be now

Yes. In particular the "heap release" bit is trying to more aggressively give memory back to the OS. We've observed in some more recent versions of tcmalloc on some (generally self-compiled, eg Arch) OSes that it will inexplicably hold a lot of free memory without it actually being in use.

no change. Should i update my tcmalloc on debian squeeze?

[: ~]# pmap x 11783|tail -n1
total kB 1762908 1051680 1047908
[: ~]# ceph -m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 26576928 ( 25.3 MB) Bytes in use by application
MALLOC: 23060480 ( 22.0 MB) Bytes free in page heap
MALLOC: 16625664 ( 15.9 MB) Bytes unmapped in page heap
MALLOC: 2737456 ( 2.6 MB) Bytes free in central cache
MALLOC: 105984 ( 0.1 MB) Bytes free in transfer cache
MALLOC: 14849200 ( 14.2 MB) Bytes free in thread caches
MALLOC: 4753 Spans in use
MALLOC: 303 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------

[: ~]# ceph m 10.255.0.100:6789 heap release
mon.a releasing free RAM back to system.
[: ~]# ceph -m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 27178352 ( 25.9 MB) Bytes in use by application
MALLOC: 4698112 ( 4.5 MB) Bytes free in page heap
MALLOC: 34320384 ( 32.7 MB) Bytes unmapped in page heap
MALLOC: 2698504 ( 2.6 MB) Bytes free in central cache
MALLOC: 165888 ( 0.2 MB) Bytes free in transfer cache
MALLOC: 14894472 ( 14.2 MB) Bytes free in thread caches
MALLOC: 4756 Spans in use
MALLOC: 303 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------

[: ~]# pmap -x 11783|tail -n1
total kB 1763004 1035780 1031172

Hmm, looks like maybe we need to send a bug to upstream (Debian and/or libgoogle-perftools devs).

Sage, any ideas who we should talk to?

The Debian Maintainer is: Daigo Moriwaki <daigo at debian.org>

Should i first try to use the one from wheezy on squeez?

yeah, try wheezy. they won't update squeeze at this point anyway.

OK backported: google-perftools from wheezy to squeeze, recompiled leveldb and ceph to reflect new google-perftools version

Will report back soon if anything has changed.

OK this def. fixes it for me. So the squeeze perftool package seems to have a memory leak.