Project

General

Profile

Actions

Bug #49846

closed

prometheus: sum_metrics not initialized

Added by Sage Weil about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-03-16T18:14:33.853+0000 7fbcb7160700  0 [prometheus ERROR root] failed to collect metrics:
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/module.py", line 230, in collect
    data = self.mod.collect()
  File "/usr/share/ceph/mgr/mgr_util.py", line 703, in wrapper
    result = f(*args, **kwargs)
  File "/usr/share/ceph/mgr/prometheus/module.py", line 1224, in collect
    self.get_collect_time_metrics()
  File "/usr/share/ceph/mgr/prometheus/module.py", line 1155, in get_collect_time_metrics
    cast(MetricCounter, sum_metric).add(duration, (method_name,))
AttributeError: 'NoneType' object has no attribute 'add'

which manifests as a failure like
2021-03-16T18:17:11.803 INFO:tasks.cephfs_test_runner:test_urls (tasks.mgr.test_prometheus.TestPrometheus) ... FAIL
2021-03-16T18:17:11.804 INFO:tasks.cephfs_test_runner:
2021-03-16T18:17:11.804 INFO:tasks.cephfs_test_runner:======================================================================
2021-03-16T18:17:11.804 INFO:tasks.cephfs_test_runner:FAIL: test_urls (tasks.mgr.test_prometheus.TestPrometheus)
2021-03-16T18:17:11.805 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-03-16T18:17:11.805 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-03-16T18:17:11.805 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_28d383a7778b37b4ae7e48d1f4a054c4b4139cb1/qa/tasks/mgr/test_prometheus.py", line 79, in test_urls
2021-03-16T18:17:11.806 INFO:tasks.cephfs_test_runner:    self.assertListEqual(failures, [])
2021-03-16T18:17:11.806 INFO:tasks.cephfs_test_runner:AssertionError: Lists differ: ['/metrics'] != []
2021-03-16T18:17:11.806 INFO:tasks.cephfs_test_runner:
2021-03-16T18:17:11.807 INFO:tasks.cephfs_test_runner:First list contains 1 additional elements.
2021-03-16T18:17:11.807 INFO:tasks.cephfs_test_runner:First extra element 0:
2021-03-16T18:17:11.808 INFO:tasks.cephfs_test_runner:'/metrics'
2021-03-16T18:17:11.808 INFO:tasks.cephfs_test_runner:
2021-03-16T18:17:11.808 INFO:tasks.cephfs_test_runner:- ['/metrics']
2021-03-16T18:17:11.808 INFO:tasks.cephfs_test_runner:+ []

/a/sage-2021-03-16_17:50:11-rados:mgr-wip-sage2-testing-2021-03-16-0838-pacific-distro-basic-smithi/5970936

Related issues 2 (1 open1 closed)

Related to mgr - Bug #46703: mgr/prometheus: introduce metric for collection timeFix Under ReviewPatrick Seidensal

Actions
Copied to mgr - Backport #49850: pacific: prometheus: sum_metrics not initializedResolvedSage WeilActions
Actions #1

Updated by Sage Weil about 3 years ago

  • Backport set to pacific
  • Pull request ID set to 40162
Actions #2

Updated by Sage Weil about 3 years ago

  • Related to Bug #46703: mgr/prometheus: introduce metric for collection time added
Actions #3

Updated by Kefu Chai about 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #4

Updated by Backport Bot about 3 years ago

  • Copied to Backport #49850: pacific: prometheus: sum_metrics not initialized added
Actions #5

Updated by Loïc Dachary almost 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions

Also available in: Atom PDF