Bug #5375
closedsqueeze tcmalloc leaks
Added by Stefan Priebe almost 11 years ago. Updated almost 11 years ago.
0%
Description
While running cuttlefish 0.61.3 or 08304a7c46da7517319b7db0b64d1c4f54771472
i'm seeing high memory usage of ceph-mon.
Freshly is starts at about 90MB now it is already at 3GB. Last week it had 12GB RAM consumed.
Updated by Sage Weil almost 11 years ago
- Subject changed from Memory leak in ceph-mon to mon: cuttlefish memory leaks
- Category set to 1
- Priority changed from High to Urgent
- Source changed from other to Community (user)
Updated by Joao Eduardo Luis almost 11 years ago
Stefan, could you please try (for your monitor's IP and PORT):
ceph -m IP:PORT heap stats ceph -m IP:PORT heap release ceph -m IP:PORT heap stats
and see if there are any significant changes?
Updated by Stefan Priebe almost 11 years ago
[: ~]# pmap -x 11783|tail -n1
total kB 1547412 688752 685152
[: ~]# ceph m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 15658448 ( 14.9 MB) Bytes in use by application
MALLOC: 33247232 ( 31.7 MB) Bytes free in page heap
MALLOC: 17707008 ( 16.9 MB) Bytes unmapped in page heap
MALLOC: 2514032 ( 2.4 MB) Bytes free in central cache
MALLOC: 157184 ( 0.1 MB) Bytes free in transfer cache
MALLOC: 14671808 ( 14.0 MB) Bytes free in thread caches
MALLOC: 4675 Spans in use
MALLOC: 305 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------
[: ~]# ceph -m 10.255.0.100:6789 heap release
mon.a releasing free RAM back to system.
[: ~]# ceph m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 16484672 ( 15.7 MB) Bytes in use by application
MALLOC: 4272128 ( 4.1 MB) Bytes free in page heap
MALLOC: 45830144 ( 43.7 MB) Bytes unmapped in page heap
MALLOC: 2543512 ( 2.4 MB) Bytes free in central cache
MALLOC: 161792 ( 0.2 MB) Bytes free in transfer cache
MALLOC: 14663464 ( 14.0 MB) Bytes free in thread caches
MALLOC: 4675 Spans in use
MALLOC: 305 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------
So actual usage (i restartet yesterday) is around 700MB. It's still raising every hour (around 30-50MB).
Updated by Stefan Priebe almost 11 years ago
not it's using 1GB should i run these commands again?
Updated by Greg Farnum almost 11 years ago
Yes. In particular the "heap release" bit is trying to more aggressively give memory back to the OS. We've observed in some more recent versions of tcmalloc on some (generally self-compiled, eg Arch) OSes that it will inexplicably hold a lot of free memory without it actually being in use.
Updated by Stefan Priebe almost 11 years ago
no change. Should i update my tcmalloc on debian squeeze?
[: ~]# pmap x 11783|tail -n1
total kB 1762908 1051680 1047908
[: ~]# ceph -m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 26576928 ( 25.3 MB) Bytes in use by application
MALLOC: 23060480 ( 22.0 MB) Bytes free in page heap
MALLOC: 16625664 ( 15.9 MB) Bytes unmapped in page heap
MALLOC: 2737456 ( 2.6 MB) Bytes free in central cache
MALLOC: 105984 ( 0.1 MB) Bytes free in transfer cache
MALLOC: 14849200 ( 14.2 MB) Bytes free in thread caches
MALLOC: 4753 Spans in use
MALLOC: 303 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------
[: ~]# ceph m 10.255.0.100:6789 heap release
mon.a releasing free RAM back to system.
[: ~]# ceph -m 10.255.0.100:6789 heap stats
mon.atcmalloc heap stats:-----------------------------------------------
MALLOC: 83955712 ( 80.1 MB) Heap size
MALLOC: 27178352 ( 25.9 MB) Bytes in use by application
MALLOC: 4698112 ( 4.5 MB) Bytes free in page heap
MALLOC: 34320384 ( 32.7 MB) Bytes unmapped in page heap
MALLOC: 2698504 ( 2.6 MB) Bytes free in central cache
MALLOC: 165888 ( 0.2 MB) Bytes free in transfer cache
MALLOC: 14894472 ( 14.2 MB) Bytes free in thread caches
MALLOC: 4756 Spans in use
MALLOC: 303 Thread heaps in use
MALLOC: 5898240 ( 5.6 MB) Metadata allocated
------------------------------------------------
[: ~]# pmap -x 11783|tail -n1
total kB 1763004 1035780 1031172
Updated by Greg Farnum almost 11 years ago
Hmm, looks like maybe we need to send a bug to upstream (Debian and/or libgoogle-perftools devs).
Sage, any ideas who we should talk to?
Updated by Stefan Priebe almost 11 years ago
The Debian Maintainer is: Daigo Moriwaki <daigo at debian.org>
Should i first try to use the one from wheezy on squeez?
Updated by Sage Weil almost 11 years ago
yeah, try wheezy. they won't update squeeze at this point anyway.
Updated by Stefan Priebe almost 11 years ago
OK backported: google-perftools from wheezy to squeeze, recompiled leveldb and ceph to reflect new google-perftools version
Will report back soon if anything has changed.
Updated by Sage Weil almost 11 years ago
- Subject changed from mon: cuttlefish memory leaks to squeeze tcmalloc leaks
- Status changed from New to 7
- Priority changed from Urgent to Normal
Updated by Stefan Priebe almost 11 years ago
OK this def. fixes it for me. So the squeeze perftool package seems to have a memory leak.
Updated by Sage Weil almost 11 years ago
- Status changed from 7 to Resolved
- Backport deleted (
cuttlefish)
i'll send a note to the email list. thanks for tracking this down!