Bug #43004
closedPrometheus Scrap error due to rbd-mirror replication stats
0%
Description
Hi !
I activate the prometheus module, but when i look on the Prometheus interface, i have that (on the Ceph Target EndPoint) :
ERROR : text format parsing error in line 311: second HELP line for metric name "ceph_rbd_mirror_replay_latency_count"
On the mgr-host when i scrap the http://mgr-host:9283/metrics , i have a result like this near the line 311 :
# HELP ceph_rbd_mirror_replay_latency_count Replay latency Count # TYPE ceph_rbd_mirror_replay_latency_count counter ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-138-disk-1"} 14493159.0 ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-167-disk-1"} 2362058.0 ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-171-disk-1"} 18046485.0 ceph_rbd_mirror_replay_latency_count{ceph_daemon="rbd-mirror.222864335",pool="cephrescue",namespace="",image="vm-109-disk-1"} 998823.0 [...]
Thanks for your help...
Files
Updated by Olivier Sauzet over 4 years ago
- File mgr-host_full-scrap.txt mgr-host_full-scrap.txt added
Konstantin Shalygin wrote:
Please, paste full scrape.
Hi,
The mgr-host host scrap from http://mgr-host:9283/metrics -> mgr-host_full-scrap.txt
Updated by Mykola Golub over 4 years ago
- Subject changed from Prometheus Scrap error to Prometheus Scrap error due to rbd-mirror replication stats
- Status changed from New to Fix Under Review
- Backport set to nautilus
- Pull request ID set to 43004
Updated by Mykola Golub over 4 years ago
The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184
@Olivier HENRY as a workaround, you could disable rbd-mirror sending its perf stats, by specifying in ceph.conf (in global or rbd-mirror's config section):
rbd_mirror_perf_stats_prio = 0
So other ceph stats could be exported to prometheus.
Updated by Dan Poltawski about 4 years ago
Mykola Golub wrote:
The problem is caused by rbd-mirror perf stats.
The fix is provided in https://github.com/ceph/ceph/pull/32184
Is there any chance this will be fixed in stable releases, the workaround is not ideal as i'd like to be monitoring of these statistics.
Updated by Mykola Golub over 2 years ago
- Status changed from Fix Under Review to Pending Backport
- Pull request ID changed from 43004 to 32184
Updated by Backport Bot over 2 years ago
- Copied to Backport #51970: nautilus: Prometheus Scrap error due to rbd-mirror replication stats added
Updated by Konstantin Shalygin over 1 year ago
- Status changed from Pending Backport to Resolved