Project

General

Profile

Actions

Bug #64970

open

mgr/dashboard: fix duplicate grafana panels when on mgr failover

Added by Avan Thakkar about 2 months ago. Updated 17 days ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Monitoring
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy, reef, squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Description of problem

here

The metrics data was not shown with "multiple matches for labels grouping labels must ensure unique matches" messages on Grafana dashboard after active mgr failover.

This is a screenshot(*1) of "ceph_pool_rd_bytes" data before failove the active mgr. The only one instance(10.74.214.226 - Active mgr then) is shown.
(*1) please see 2024-02-07 14-47-25_before_failover_the_mgr.png

But after active mgr failover, 2 instances (10.74.214.226 and mons-2.cephtest61.lab.psi.pnq2.redhat.com - Current Active mgr) are shown(*2)
(*2) please see 2024-02-07 14-47-32_after_failover_the_mgr.png

It seems there are 2 instances with same metrics collected by ceph-mgr after mgr failover
(one is the metrics collected by previous active mgr, Another is the metrics currently collecting by current active mgr)
Grafana tried to display it, but It failed with "multiple matches for labels grouping labels must ensure unique matches" message.


Files

Screenshot from 2024-03-18 23-12-14.png (215 KB) Screenshot from 2024-03-18 23-12-14.png after_failover_the_mgr Avan Thakkar, 03/18/2024 05:43 PM
Screenshot from 2024-03-18 23-12-47.png (313 KB) Screenshot from 2024-03-18 23-12-47.png before_failover_the_mgr Avan Thakkar, 03/18/2024 05:43 PM
Screenshot from 2024-03-18 23-12-07.png (200 KB) Screenshot from 2024-03-18 23-12-07.png multiple matches issue grafan panel Avan Thakkar, 03/18/2024 05:43 PM

Related issues 3 (3 open0 closed)

Copied to Dashboard - Backport #65512: reef: mgr/dashboard: fix duplicate grafana panels when on mgr failoverIn ProgressAdam KingActions
Copied to Dashboard - Backport #65513: quincy: mgr/dashboard: fix duplicate grafana panels when on mgr failoverIn ProgressAdam KingActions
Copied to Dashboard - Backport #65515: squid: mgr/dashboard: fix duplicate grafana panels when on mgr failoverIn ProgressAdam KingActions
Actions #1

Updated by Avan Thakkar about 1 month ago

  • Backport changed from quincy, reef to quincy, reef, squid
Actions #2

Updated by Avan Thakkar about 1 month ago

  • Pull request ID set to 56626
Actions #3

Updated by Adam King 17 days ago

  • Status changed from Fix Under Review to Pending Backport
Actions #4

Updated by Backport Bot 17 days ago

  • Copied to Backport #65512: reef: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions #5

Updated by Backport Bot 17 days ago

  • Copied to Backport #65513: quincy: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions #6

Updated by Backport Bot 17 days ago

  • Copied to Backport #65515: squid: mgr/dashboard: fix duplicate grafana panels when on mgr failover added
Actions #7

Updated by Backport Bot 17 days ago

  • Tags set to backport_processed
Actions

Also available in: Atom PDF