Project

General

Profile

Feature #24426

mds: add second level cache backed by local SSD or NVRAM

Added by Patrick Donnelly 7 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
06/05/2018
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Reviewed:
Affected Versions:
Component(FS):
MDS
Labels (FS):
task(hard)
Pull request ID:

Description

Idea is to have a second level to the MDS cache to improve access time and reduce reads on the metadata pool. This would be a volatile cache and be rebuilt on MDS restart.

The implementation for this could be very simple by mmap'ing the backing (device-as-)file and allocating metadata objects against that single memory extent. Then, let the kernel do the first level cache management. Most of the annoying work of allocating MDS cache objects through a custom allocator is already done with the recent mempool changes (to track the size of the MDS cache). However, a new allocator will be required as mempools funnel everything through the default allocator.

An alternate and potentially better solution would be to manually handle the second-level cache because having 512GB+ MDS caches may have non-trivial performance costs (e.g. having so many objects in various lists systemically slows down the MDS). The implementation also would not require mmap/allocator voodoo.

Note: most metadata pools already are SSD backed so this saves us a network round-trip and RADOS overhead. A performance comparison would be interesting afterwards.

Also available in: Atom PDF