Project

General

Profile

Bug #48354

Logic error in default prom alert 'pool filling up'

Added by Daniël Vos over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus, nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hey guys,

The default alert 'pool filling up' will always be firing if your pool if filled for more than 50%.

Referring to this specific alert: https://github.com/ceph/ceph/blob/ac33cea383f03f34000c83073b6ae331941c0431/monitoring/prometheus/alerts/ceph_default_alerts.yml#L230

Example:
My pool with a capacity of 21TB has 13.5TB data and 7.5TB storage available

ceph_pool_stored[2d], 3600 * 24 * 5

has a value of 13.5TB
ceph_pool_max_avail

has a value of 7.5TB

Therefore `ceph_pool_max_avail` is always smaller than `ceph_pool_stored[2d], 3600 * 24 * 5` which is why my alert has been firing for weeks now.


Related issues

Copied to mgr - Backport #48578: octopus: Logic error in default prom alert 'pool filling up' Resolved
Copied to mgr - Backport #48957: nautilus: Logic error in default prom alert 'pool filling up' Resolved

History

#1 Updated by Daniël Vos over 3 years ago

I've created a PR to tackle this issue. https://github.com/ceph/ceph/pull/38282

#2 Updated by Volker Theile over 3 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 38282

#3 Updated by Volker Theile over 3 years ago

Hi Daniël, do you want to see your fix in Octopus, too? If yes, we need to set the backport field here.

#4 Updated by Patrick Seidensal over 3 years ago

  • Backport set to octopus

#5 Updated by Daniël Vos over 3 years ago

Volker Theile wrote:

Hi Daniël, do you want to see your fix in Octopus, too? If yes, we need to set the backport field here.

Patrick has made the decision for me :-)

#6 Updated by Kefu Chai over 3 years ago

  • Status changed from Fix Under Review to Pending Backport

#7 Updated by Backport Bot over 3 years ago

  • Copied to Backport #48578: octopus: Logic error in default prom alert 'pool filling up' added

#8 Updated by Ernesto Puerta about 3 years ago

  • Status changed from Pending Backport to Resolved

#9 Updated by Ernesto Puerta about 3 years ago

  • Status changed from Resolved to Pending Backport

#10 Updated by Ernesto Puerta about 3 years ago

  • Backport changed from octopus to octopus, nautilus

#11 Updated by Backport Bot about 3 years ago

  • Copied to Backport #48957: nautilus: Logic error in default prom alert 'pool filling up' added

#12 Updated by Konstantin Shalygin about 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF