Project

General

Profile

Bug #45311

Feature #39478: mgr/dashboard: new RGW workflows & RGW enhancements

Feature #39494: mgr/dashboard: Add overview landing page for RGW

Feature #39369: mgr/dashboard: show RGW multi-site sync status info

rgw: provide right format for rgw sync perf. counters.

Added by Alfonso Martínez 9 months ago. Updated 9 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
dashboard/rgw
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Currently, prometheus mgr module gets rgw perf counters as in https://tracker.ceph.com/attachments/download/4854/rgw_perf_counters.txt

In that file, the zone name used is: zone2_zg1_realm1, zone3-zg1-realm1, zone2-zg1-realm2.

After standard transformation for prometheus server consumption, this is how it arrives to prometheus server:

ceph_data_sync_from_<ZONE_NAME>_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph"}

1) The zone name should be added as label (e.g. zone="<ZONE_NAME>") (as it is required for metric consumption purposes).
2) The metric name format should be fixed (i.e. it should not contain any particular detail: zone, zonegroup, realm, ...).
It should be:

ceph_rgw_sync_from_zone_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph",zone="<ZONE_NAME>"}

The issue is similar to this:
https://tracker.ceph.com/issues/39977

rgw_perf_counters.txt View (5.99 KB) Alfonso Martínez, 04/28/2020 02:53 PM


Related issues

Related to mgr - Feature #45310: mgr/dashboard: add grafana dashboards for rgw multisite sync info Resolved

History

#1 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)

#2 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)

#3 Updated by Alfonso Martínez 9 months ago

  • Related to Feature #45310: mgr/dashboard: add grafana dashboards for rgw multisite sync info added

#4 Updated by Alfonso Martínez 9 months ago

  • Description updated (diff)

#5 Updated by Alfonso Martínez 9 months ago

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:

{'data-sync-from-zone2-zg1-realm1.fetch_bytes': {'description': 'Number of object bytes replicated', 'type': 6, 'priority': 5, 'units': 1, 'value': 0, 'count': 0}

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

#6 Updated by Casey Bodley 9 months ago

Alfonso Martínez wrote:

2) The metric name format should be fixed (i.e. it should not contain any particular detail: zone, zonegroup, realm, ...).
It should be:
[...]

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from. a radosgw in zoneA may be replicating from several other zones, and needs to track 'data-sync-from-zoneB' separately from 'data-sync-from-zoneC', etc.

#7 Updated by Casey Bodley 9 months ago

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

#8 Updated by Alfonso Martínez 9 months ago

Casey Bodley wrote:

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

OK, the important thing here is to provide a fixed metric name.

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from.

I know, the proposal is to put the source zone name in a label and keep the metric name fixed, so prometheus server receives sth like:

ceph_data_sync_from_zone_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph",zone="<ZONE_REPLICATING_FROM>"}

#9 Updated by Casey Bodley 9 months ago

Alfonso Martínez wrote:

Casey Bodley wrote:

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

OK, the important thing here is to provide a fixed metric name.

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from.

I know, the proposal is to put the source zone name in a label and keep the metric name fixed, so prometheus server receives sth like:
[...]

okay, gotcha. labels aren't part of ceph's PerfCounters api, so radosgw can't send it that way. i assume that ceph-mgr is adding those other labels somewhere down the line, and it should be able to transform the "data-sync-from-<zone>" names into the format you're expecting for prometheus

Also available in: Atom PDF