Project

General

Profile

Bug #43004

Prometheus Scrap error due to rbd-mirror replication stats

Added by Olivier Sauzet almost 2 years ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi !

I activate the prometheus module, but when i look on the Prometheus interface, i have that (on the Ceph Target EndPoint) :

ERROR : text format parsing error in line 311: second HELP line for metric name "ceph_rbd_mirror_replay_latency_count" 

On the mgr-host when i scrap the http://mgr-host:9283/metrics , i have a result like this near the line 311 :

# HELP ceph_rbd_mirror_replay_latency_count Replay latency Count
# TYPE ceph_rbd_mirror_replay_latency_count counter
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-138-disk-1"} 14493159.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-167-disk-1"} 2362058.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-171-disk-1"} 18046485.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-109-disk-1"} 998823.0
[...]

Thanks for your help...

Prometheus_error.png View - error on prometheus (25 KB) Olivier Sauzet, 11/25/2019 10:27 AM

mgr-host_full-scrap.txt View (128 KB) Olivier Sauzet, 11/26/2019 07:14 AM


Related issues

Copied to mgr - Backport #51970: nautilus: Prometheus Scrap error due to rbd-mirror replication stats In Progress

History

#1 Updated by Konstantin Shalygin almost 2 years ago

Please, paste full scrape.

#2 Updated by Olivier Sauzet almost 2 years ago

Konstantin Shalygin wrote:

Please, paste full scrape.

Hi,
The mgr-host host scrap from http://mgr-host:9283/metrics -> mgr-host_full-scrap.txt

#3 Updated by Mykola Golub almost 2 years ago

  • Assignee set to Mykola Golub

#4 Updated by Mykola Golub almost 2 years ago

  • Subject changed from Prometheus Scrap error to Prometheus Scrap error due to rbd-mirror replication stats
  • Status changed from New to Fix Under Review
  • Backport set to nautilus
  • Pull request ID set to 43004

#5 Updated by Mykola Golub almost 2 years ago

The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184

@Olivier as a workaround, you could disable rbd-mirror sending its perf stats, by specifying in ceph.conf (in global or rbd-mirror's config section):

rbd_mirror_perf_stats_prio = 0

So other ceph stats could be exported to prometheus.

#6 Updated by Dan Poltawski over 1 year ago

Mykola Golub wrote:

The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184

Is there any chance this will be fixed in stable releases, the workaround is not ideal as i'd like to be monitoring of these statistics.

#7 Updated by Mykola Golub about 2 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Pull request ID changed from 43004 to 32184

#8 Updated by Backport Bot about 2 months ago

  • Copied to Backport #51970: nautilus: Prometheus Scrap error due to rbd-mirror replication stats added

Also available in: Atom PDF