Project

General

Profile

Actions

Feature #52414

closed

mgr/prometheus: offer ability to disable cache

Added by Patrick Seidensal over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Category:
Monitoring
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus, pacific
Reviewed:
Affected Versions:
Pull request ID:

Description

Offer ability to disable the cache

The Prometheus mgr module gathers the data for the whole Ceph cluster. Contrary to how Prometheus' exporters are usually designed, we only have one exporter for Ceph specific data. This increases the time it takes to gather the data, as one host is required to collect all of it.

At some point, we've had so many issues with the time it takes to collect the data (clusters with around 1000 OSDs), that it exceeded the 10 or even 15 seconds scrape interval of Prometheus. Considering that in a high available environment, there's likely to more than one Prometheus instance to scrape data from the Prometheus mgr module, implementing a cache was necessary.

This cache is enabled by default and up to now, cannot be disabled. While using this cache is highly effective in mitigating any issues with collecting the data, it is not strictly required for smaller (or faster) deployments. It actually is better if the cache is not used.

But since the cache was introduced, there hasn't been a possibility to disable it. To ease debugging issues but also be able to permanently disable the cache, we need to implement a switch to turn it off.


Related issues 3 (0 open3 closed)

Copied to Dashboard - Backport #52526: octopus: mgr/prometheus: offer ability to disable cacheRejectedActions
Copied to Dashboard - Backport #52527: nautilus: mgr/prometheus: offer ability to disable cacheRejectedActions
Copied to Dashboard - Backport #52528: pacific: mgr/prometheus: offer ability to disable cacheResolvedErnesto PuertaActions
Actions #1

Updated by Patrick Seidensal over 2 years ago

  • Subject changed from mgr/prometheus: offer ability to disable cache mgr/dashboard: short_description to mgr/prometheus: offer ability to disable cache
Actions #2

Updated by Patrick Seidensal over 2 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 42933
Actions #3

Updated by Kefu Chai over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #4

Updated by Backport Bot over 2 years ago

  • Copied to Backport #52526: octopus: mgr/prometheus: offer ability to disable cache added
Actions #5

Updated by Backport Bot over 2 years ago

  • Copied to Backport #52527: nautilus: mgr/prometheus: offer ability to disable cache added
Actions #6

Updated by Backport Bot over 2 years ago

  • Copied to Backport #52528: pacific: mgr/prometheus: offer ability to disable cache added
Actions #7

Updated by Ernesto Puerta over 2 years ago

  • Status changed from Pending Backport to Resolved
Actions #8

Updated by Konstantin Shalygin about 2 years ago

  • Backport changed from nautilus, octopus, pacific to octopus, pacific
Actions

Also available in: Atom PDF