Ceph » CephFS

Idea is to have a second level to the MDS cache to improve access time and reduce reads on the metadata pool. This would be a volatile cache and be rebuilt on MDS restart.

The implementation for this could be very simple by mmap'ing the backing (device-as-)file and allocating metadata objects against that single memory extent. Then, let the kernel do the first level cache management. Most of the annoying work of allocating MDS cache objects through a custom allocator is already done with the recent mempool changes (to track the size of the MDS cache). However, a new allocator will be required as mempools funnel everything through the default allocator.

An alternate and potentially better solution would be to manually handle the second-level cache because having 512GB+ MDS caches may have non-trivial performance costs (e.g. having so many objects in various lists systemically slows down the MDS). The implementation also would not require mmap/allocator voodoo.

Note: most metadata pools already are SSD backed so this saves us a network round-trip and RADOS overhead. A performance comparison would be interesting afterwards.