Ceph » rbd

The rbd cache uses the data extent size when determining the space utilized by the cache. When writing data in chunks smaller than the page size, the wasted heap space is not accounted for in the cache sizing. The smaller the block size, the the larger the cache (heap) can grow beyond the original max size constraint.

Below is a massif snippet writing 4K blocks using the 32/24/16 defaults for max cache, max dirty, and target dirty.

--------------------------------------------------------------------------------
n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
81 3,765,049,024 112,570,288 79,148,293 33,421,995 0
70.31% (79,148,293B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->58.20% (65,519,616B) 0x59C0BFE: ceph::buffer::create_page_aligned(unsigned int) (buffer.cc:239) | ->58.20% (65,519,616B) 0x59C0D97: ceph::buffer::list::append(char const*, unsigned int) (buffer.cc:1242) | | ->58.19% (65,503,232B) 0x54ABD70: librbd::aio_write(librbd::ImageCtx*, unsigned long, unsigned long, char const*, librbd::AioCompletion*) (internal.cc:2949) | | | ->58.19% (65,503,232B) 0x40D265: do_bench_write(librbd::Image&, unsigned long, unsigned long, unsigned long, std::string) (rbd.cc:852) | | | ->58.19% (65,503,232B) 0x416204: main (rbd.cc:2779)