Project

General

Profile

Actions

Bug #43004

closed

Prometheus Scrap error due to rbd-mirror replication stats

Added by Olivier Sauzet over 4 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi !

I activate the prometheus module, but when i look on the Prometheus interface, i have that (on the Ceph Target EndPoint) :

ERROR : text format parsing error in line 311: second HELP line for metric name "ceph_rbd_mirror_replay_latency_count" 

On the mgr-host when i scrap the http://mgr-host:9283/metrics , i have a result like this near the line 311 :

# HELP ceph_rbd_mirror_replay_latency_count Replay latency Count
# TYPE ceph_rbd_mirror_replay_latency_count counter
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-138-disk-1"} 14493159.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-167-disk-1"} 2362058.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-171-disk-1"} 18046485.0
ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-109-disk-1"} 998823.0
[...]

Thanks for your help...


Files

Prometheus_error.png (25 KB) Prometheus_error.png error on prometheus Olivier Sauzet, 11/25/2019 10:27 AM
mgr-host_full-scrap.txt (128 KB) mgr-host_full-scrap.txt Olivier Sauzet, 11/26/2019 07:14 AM

Related issues 1 (0 open1 closed)

Copied to mgr - Backport #51970: nautilus: Prometheus Scrap error due to rbd-mirror replication statsRejectedMykola GolubActions
Actions #1

Updated by Konstantin Shalygin over 4 years ago

Please, paste full scrape.

Actions #2

Updated by Olivier Sauzet over 4 years ago

Konstantin Shalygin wrote:

Please, paste full scrape.

Hi,
The mgr-host host scrap from http://mgr-host:9283/metrics -> mgr-host_full-scrap.txt

Actions #3

Updated by Mykola Golub over 4 years ago

  • Assignee set to Mykola Golub
Actions #4

Updated by Mykola Golub over 4 years ago

  • Subject changed from Prometheus Scrap error to Prometheus Scrap error due to rbd-mirror replication stats
  • Status changed from New to Fix Under Review
  • Backport set to nautilus
  • Pull request ID set to 43004
Actions #5

Updated by Mykola Golub over 4 years ago

The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184

@Olivier HENRY as a workaround, you could disable rbd-mirror sending its perf stats, by specifying in ceph.conf (in global or rbd-mirror's config section):

rbd_mirror_perf_stats_prio = 0

So other ceph stats could be exported to prometheus.

Actions #6

Updated by Dan Poltawski about 4 years ago

Mykola Golub wrote:

The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184

Is there any chance this will be fixed in stable releases, the workaround is not ideal as i'd like to be monitoring of these statistics.

Actions #7

Updated by Mykola Golub over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Pull request ID changed from 43004 to 32184
Actions #8

Updated by Backport Bot over 2 years ago

  • Copied to Backport #51970: nautilus: Prometheus Scrap error due to rbd-mirror replication stats added
Actions #9

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions #10

Updated by Konstantin Shalygin over 1 year ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF