Project

General

Profile

Feature #52414

mgr/prometheus: offer ability to disable cache

Added by Patrick Seidensal 2 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Normal
Category:
Monitoring
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus, octopus, pacific
Reviewed:
Affected Versions:
Pull request ID:

Description

Offer ability to disable the cache

The Prometheus mgr module gathers the data for the whole Ceph cluster. Contrary to how Prometheus' exporters are usually designed, we only have one exporter for Ceph specific data. This increases the time it takes to gather the data, as one host is required to collect all of it.

At some point, we've had so many issues with the time it takes to collect the data (clusters with around 1000 OSDs), that it exceeded the 10 or even 15 seconds scrape interval of Prometheus. Considering that in a high available environment, there's likely to more than one Prometheus instance to scrape data from the Prometheus mgr module, implementing a cache was necessary.

This cache is enabled by default and up to now, cannot be disabled. While using this cache is highly effective in mitigating any issues with collecting the data, it is not strictly required for smaller (or faster) deployments. It actually is better if the cache is not used.

But since the cache was introduced, there hasn't been a possibility to disable it. To ease debugging issues but also be able to permanently disable the cache, we need to implement a switch to turn it off.


Related issues

Copied to Dashboard - Backport #52526: octopus: mgr/prometheus: offer ability to disable cache New
Copied to Dashboard - Backport #52527: nautilus: mgr/prometheus: offer ability to disable cache New
Copied to Dashboard - Backport #52528: pacific: mgr/prometheus: offer ability to disable cache New

History

#1 Updated by Patrick Seidensal 2 months ago

  • Subject changed from mgr/prometheus: offer ability to disable cache mgr/dashboard: short_description to mgr/prometheus: offer ability to disable cache

#2 Updated by Patrick Seidensal 2 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 42933

#3 Updated by Kefu Chai about 2 months ago

  • Status changed from Fix Under Review to Pending Backport

#4 Updated by Backport Bot about 2 months ago

  • Copied to Backport #52526: octopus: mgr/prometheus: offer ability to disable cache added

#5 Updated by Backport Bot about 2 months ago

  • Copied to Backport #52527: nautilus: mgr/prometheus: offer ability to disable cache added

#6 Updated by Backport Bot about 2 months ago

  • Copied to Backport #52528: pacific: mgr/prometheus: offer ability to disable cache added

Also available in: Atom PDF