Project

General

Profile

Actions
Copy link

Bug #48354

closed

Logic error in default prom alert 'pool filling up'

Added by Daniël Vos over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus, nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v15.2.6
ceph-qa-suite:
Pull request ID:
38282
Crash signature (v1):
Crash signature (v2):

Description

Hey guys,

The default alert 'pool filling up' will always be firing if your pool if filled for more than 50%.

Referring to this specific alert: https://github.com/ceph/ceph/blob/ac33cea383f03f34000c83073b6ae331941c0431/monitoring/prometheus/alerts/ceph_default_alerts.yml#L230

Example:
My pool with a capacity of 21TB has 13.5TB data and 7.5TB storage available

ceph_pool_stored[2d], 3600 * 24 * 5

has a value of 13.5TB 
ceph_pool_max_avail

has a value of 7.5TB

Therefore `ceph_pool_max_avail` is always smaller than `ceph_pool_stored[2d], 3600 * 24 * 5` which is why my alert has been firing for weeks now.

Related issues 2 (0 open — 2 closed)

Copied to mgr - Backport #48578: octopus: Logic error in default prom alert 'pool filling up'ResolvedLaura PaduanoActions
Copied to mgr - Backport #48957: nautilus: Logic error in default prom alert 'pool filling up'ResolvedKonstantin ShalyginActions
Actions
Copy link
 #1

Updated by Daniël Vos over 3 years ago

I've created a PR to tackle this issue. https://github.com/ceph/ceph/pull/38282

Actions
Copy link
 #2

Updated by Volker Theile over 3 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 38282
Actions
Copy link
 #3

Updated by Volker Theile over 3 years ago

Hi Daniël, do you want to see your fix in Octopus, too? If yes, we need to set the backport field here.

Actions
Copy link
 #4

Updated by Patrick Seidensal over 3 years ago

  • Backport set to octopus
Actions
Copy link
 #5

Updated by Daniël Vos over 3 years ago

Volker Theile wrote:

Hi Daniël, do you want to see your fix in Octopus, too? If yes, we need to set the backport field here.

Patrick has made the decision for me :-)

Actions
Copy link
 #6

Updated by Kefu Chai over 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #7

Updated by Backport Bot over 3 years ago

  • Copied to Backport #48578: octopus: Logic error in default prom alert 'pool filling up' added
Actions
Copy link
 #8

Updated by Ernesto Puerta over 3 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #9

Updated by Ernesto Puerta over 3 years ago

  • Status changed from Resolved to Pending Backport
Actions
Copy link
 #10

Updated by Ernesto Puerta over 3 years ago

  • Backport changed from octopus to octopus, nautilus
Actions
Copy link
 #11

Updated by Backport Bot over 3 years ago

  • Copied to Backport #48957: nautilus: Logic error in default prom alert 'pool filling up' added
Actions
Copy link
 #12

Updated by Konstantin Shalygin over 3 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF