Bug #45311: rgw: provide right format for rgw sync perf. counters. - Dashboard - Ceph

Actions

Copy link

Bug #45311

closed

Feature #39478: mgr/dashboard: new RGW workflows & RGW enhancements

Feature #39494: mgr/dashboard: Add overview landing page for RGW

Feature #39369: mgr/dashboard: show RGW multi-site sync status info

rgw: provide right format for rgw sync perf. counters.

Added by Alfonso Martínez almost 4 years ago. Updated almost 3 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Component - RGW

Target version:

Ceph - v16.0.0

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Currently, prometheus mgr module gets rgw perf counters as in https://tracker.ceph.com/attachments/download/4854/rgw_perf_counters.txt

In that file, the zone name used is: zone2_zg1_realm1, zone3-zg1-realm1, zone2-zg1-realm2.

After standard transformation for prometheus server consumption, this is how it arrives to prometheus server:

ceph_data_sync_from_<ZONE_NAME>_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph"}

1) The zone name should be added as label (e.g. zone="<ZONE_NAME>") (as it is required for metric consumption purposes).
2) The metric name format should be fixed (i.e. it should not contain any particular detail: zone, zonegroup, realm, ...).
It should be:

ceph_rgw_sync_from_zone_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph",zone="<ZONE_NAME>"}

The issue is similar to this:
https://tracker.ceph.com/issues/39977

Files

rgw_perf_counters.txt (5.99 KB) rgw_perf_counters.txt

Alfonso Martínez, 04/28/2020 02:53 PM

Related issues 2 (1 open — 1 closed)

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Description updated (diff)

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Description updated (diff)

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Related to Feature #45310: mgr/dashboard: add grafana dashboards for rgw multisite sync info added

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Description updated (diff)

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:

{'data-sync-from-zone2-zg1-realm1.fetch_bytes': {'description': 'Number of object bytes replicated', 'type': 6, 'priority': 5, 'units': 1, 'value': 0, 'count': 0}

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

Actions

Copy link

Updated by Casey Bodley almost 4 years ago

Alfonso Martínez wrote:

2) The metric name format should be fixed (i.e. it should not contain any particular detail: zone, zonegroup, realm, ...).
It should be:
[...]

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from. a radosgw in zoneA may be replicating from several other zones, and needs to track 'data-sync-from-zoneB' separately from 'data-sync-from-zoneC', etc.

Actions

Copy link

Updated by Casey Bodley almost 4 years ago

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

Actions

Copy link

Updated by Alfonso Martínez almost 4 years ago

Casey Bodley wrote:

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

OK, the important thing here is to provide a fixed metric name.

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from.

I know, the proposal is to put the source zone name in a label and keep the metric name fixed, so prometheus server receives sth like:

ceph_data_sync_from_zone_fetch_bytes_count{ceph_daemon="rgw.8000",instance="ceph-rpm:9283",job="ceph",zone="<ZONE_REPLICATING_FROM>"}

Actions

Copy link

Updated by Casey Bodley almost 4 years ago

Alfonso Martínez wrote:

Casey Bodley wrote:

Alfonso Martínez wrote:

Hi @CaseyBodley,

Provided that from C++ side it comes in the format:
[...]

Is it possible that the metric has the name format: {'rgw-sync-from-zone.fetch_bytes':
and with a "labels" field with label added: zone = "zone2-zg1-realm1" (example) ?

the name format from radosgw's perspective is "data-sync-from-<source zone>"

OK, the important thing here is to provide a fixed metric name.

note that the zone name in "data-sync-from-<zone>" is not the same as the zone that radosgw is running in, it's the source zone that it's replicating from.

I know, the proposal is to put the source zone name in a label and keep the metric name fixed, so prometheus server receives sth like:
[...]

okay, gotcha. labels aren't part of ceph's PerfCounters api, so radosgw can't send it that way. i assume that ceph-mgr is adding those other labels somewhere down the line, and it should be able to transform the "data-sync-from-<zone>" names into the format you're expecting for prometheus

Actions

Copy link

#10