Ceph

We observed high memory consumption by OSD daemons when we evaluating Hammer (0.94.2) in Yahoo! environment. Initially memory consumption per OSD was only about 1G (or even less) while it grow to almost 3G after 2 hours load test, which caused the system frozen/OOM (10 OSD run on one host with 32G physical memory) and finally some OSD processes were killed by system.

Below is the heap dump, almost 1.9G memory was occupied by heap freelist.

$ sudo ceph tell osd.32 heap stats
osd.32 tcmalloc heap stats:------------------------------------------------
MALLOC: 1217475232 ( 1161.1 MiB) Bytes in use by application
MALLOC: + 2031304704 ( 1937.2 MiB) Bytes in page heap freelist
MALLOC: + 49365752 ( 47.1 MiB) Bytes in central cache freelist
MALLOC: + 4159488 ( 4.0 MiB) Bytes in transfer cache freelist
MALLOC: + 56619624 ( 54.0 MiB) Bytes in thread cache freelists
MALLOC: + 14528664 ( 13.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 3373453464 ( 3217.2 MiB) Actual memory used (physical + swap)
MALLOC: + 4677632 ( 4.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 3378131096 ( 3221.6 MiB) Virtual address space used
MALLOC:
MALLOC: 135722 Spans in use
MALLOC: 1152 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------

After run "heap release" the 1.9G memory was able to be released to system.

$ sudo ceph tell osd.32 heap release
osd.32 releasing free RAM back to system.

$ sudo ceph tell osd.32 heap stats
osd.32 tcmalloc heap stats:------------------------------------------------
MALLOC: 1211800264 ( 1155.7 MiB) Bytes in use by application
MALLOC: + 286720 ( 0.3 MiB) Bytes in page heap freelist
MALLOC: + 54191184 ( 51.7 MiB) Bytes in central cache freelist
MALLOC: + 7419648 ( 7.1 MiB) Bytes in transfer cache freelist
MALLOC: + 54217192 ( 51.7 MiB) Bytes in thread cache freelists
MALLOC: + 14528664 ( 13.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 1342443672 ( 1280.3 MiB) Actual memory used (physical + swap)
MALLOC: + 2035687424 ( 1941.4 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 3378131096 ( 3221.6 MiB) Virtual address space used
MALLOC:
MALLOC: 135729 Spans in use
MALLOC: 1148 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------