Ceph » RADOS

When trying to cache object content BlueStore uses twice as much memory than it really caches.

The root cause for that seems to be in buffer::create_page_aligned implementation. Actually it results in
new raw_posix_aligned() -> mempool::buffer_data::alloc_char.allocate_aligned(len, align) -> posix_memalign((void**)(void*)&ptr, align, total);
sequence that in fact does 2 allocations:
1) for raw_posix_aligned
2) for data itself.

It looks like this sequence causes 2 * 4096 bytes allocation instead of 4096 + small_delta. See a code snipped attached.
The additional trick is that mempool stuff is unable to estimate an overhead and hence BlueStore cache cleanup doesn't work properly.

The issue is reproducible under Ubuntu 16.04 for both jemalloc and tcmalloc builds.

Output for 16Gb allocation:

Mem before: VmRSS: 45232 kB
Mem after: VmRSS: 33599524 kB
Mem actually used: 33554292 kB
Mem pool reports: 16777216 kB
Mem before2: VmRSS: 2161412 kB
Mem after2: VmRSS: 33632268 kB
Mem actually used: 32226156544 bytes