Project

General

Profile

Actions
Copy link

Bug #44590

closed

prometheus metrics wrongly reports scrubbing pgs

Added by Jacek S. about 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic,nautilus,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.8
ceph-qa-suite:
Pull request ID:
33967
Crash signature (v1):
Crash signature (v2):

Description

After change of ceph_pg_* in ceph 14.2.8 metrics ceph_pg_active and ceph_pg_clean do not count groups that are scrubbing
So cluster with all active and clean groups but with 3 scrubs pending will have:

ceph_pg_total{pool_id="26"} 2048.0
ceph_pg_active{pool_id="26"} 2044.0
ceph_pg_clean{pool_id="26"} 2044.0
ceph_pg_scrubbing{pool_id="26"} 0.0

while:

ceph pg dump|grep "^26\."|awk '{ print $12; }' | sort|uniq -c
   2044 active+clean
      2 active+clean+scrubbing
      2 active+clean+scrubbing+deep

Related issues 3 (0 open — 3 closed)

Copied to mgr - Backport #44735: nautilus: prometheus metrics wrongly reports scrubbing pgsResolvedKonstantin ShalyginActions
Copied to mgr - Backport #44736: mimic: prometheus metrics wrongly reports scrubbing pgsRejectedShyukri ShyukrievActions
Copied to mgr - Backport #44737: octopus: prometheus metrics wrongly reports scrubbing pgsResolvedKonstantin ShalyginActions
Actions
Copy link
 #1

Updated by Jacek S. about 4 years ago

Durign a debug I found that for each pool each kind of state is overwriting a previous, I'm preparing a MR

Actions
Copy link
 #3

Updated by Neha Ojha about 4 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 33967
Actions
Copy link
 #4

Updated by Kefu Chai about 4 years ago

  • Backport set to mimic,nautilus
Actions
Copy link
 #5

Updated by Jonas Jelten about 4 years ago

This probably also causes my bug that when some pgs are degraded and active, they're not reported as active in prometheus.

Actions
Copy link
 #6

Updated by Kefu Chai about 4 years ago

  • Backport changed from mimic,nautilus to mimic,nautilus,octopus
Actions
Copy link
 #7

Updated by Sage Weil about 4 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #8

Updated by Konstantin Shalygin about 4 years ago

  • Copied to Backport #44735: nautilus: prometheus metrics wrongly reports scrubbing pgs added
Actions
Copy link
 #9

Updated by Konstantin Shalygin about 4 years ago

  • Copied to Backport #44736: mimic: prometheus metrics wrongly reports scrubbing pgs added
Actions
Copy link
 #10

Updated by Konstantin Shalygin about 4 years ago

  • Copied to Backport #44737: octopus: prometheus metrics wrongly reports scrubbing pgs added
Actions
Copy link
 #11

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF