Summary¶

In order to reduce the chances of mgr modules and, specifically, dashboard to compromise the Ceph cluster performance by an increase in the frequency of API calls (unlike other mgr Modules, Dashboard load is predominantly user-driven), a multi-layered caching approach should be put in place.

Current status¶

Existing caching approaches and issues:

• Ceph-mgr itself caches a lot of API calls (get_module_option, get, get_server, get_metadata, ), so not every request to the ceph-mgr API hits the Ceph cluster. However, the send_command() is not cached and might have a performance impact.
• Additionally, one bottleneck in ceph-mgr is the PyFormatter, the class responsible for deserializing C++ binary structs to Python objects. For big objects (osd_map) this deserialization is not negligible, so it might be worthy to explore caching the resulting deserialized Python object or explore an incremental approach that doesn't involve processing the same data time after time.
• Dashboard-backend: ViewCache decouples REST controller request from ceph-mgr API ones and allows for asynchronous fetching of data.

The following picture shows the existing approaches (ViewCache) and the ones to explore, as a well as other potential points (PyFormatter and non-cached send_command):

Proposal¶

Layers:

• Ceph-mgr API:
  • C++: this is optimal, as the cached data is shared across modules. However is less trivial to implement.
  • Python: cachetools. This one could be introduced at per-module level (every module interacts with each version of the cached ceph-mgr API methods) or shared (all modules consume the cached versions of the ceph-mgr API methods, although this could bring issues with modules modifying the objects returned by the cached methods).
• Dashboard back-end:
  • Python: cachetools
  • CherryPy Cache (it also takes care of the HTTP caching)
• Dashboard front-end:
  • HTTP Cache (browser provided?)
  • Typescript/JS: memoize-cache-decorator or ts-cacheable

Pros:

• Reduced load in ceph-mgr
• Shorter response times

Cons:

• Increased memory usage
• Stale data (while TTL caches can improve this)
• Data serializations issues
• Leaks/ref counting issues

Implementation details¶

As caching is an optimization strategy and optimization needs to be benchmarked, the first step would be to implement a way to measure the effectiveness of caching:

• From the backend/inner side of the system, that could be the number of calls that actually hit the ceph-mgr API (log message, new mgr CLI command, ...)
• From the user facing side of the system, that could be the latency of the call (cold, when cache is not populated, vs. warm cache, when cache is populated and is hit by the request). Here there 2 possible user-facing points: direct RESTful interface (via curl) and WebUI (via Angular/JS). The second one is more realistic but less finegrained, so the 1st one would be preferred for quick benchmarking.

Additionally, it would be interesting to short list the quick-wins, those ceph-mgr API calls and dashboard RESTful API endpoints that would benefit most from this caching scheme: those performing more calls to the ceph-mgr API. Keep in mind that the number of calls might depend on the scale of the cluster (e.g.: GET /rbd might required 1-many calls per RBD image created, and there are some users with thousands of RBD images).