Actions
Bug #63292
openPrometheus exporter bug
Status:
New
Priority:
Normal
Assignee:
-
Category:
ceph-mgr
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
After upgrading from Quincy to Reef the prometheus exporter stopped working. The /metrics endpoint now returns 503 with error message:
Traceback (most recent call last): File "/lib/python3/dist-packages/cherrypy/_cprequest.py", line 670, in respond response.body = self.handler() File "/lib/python3/dist-packages/cherrypy/lib/encoding.py", line 220, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1816, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1827, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet')
In the mgr logs we see:
2023-10-23T11:32:05.351-0400 7fee7e97b700 0 [prometheus ERROR root] failed to collect metrics: Traceback (most recent call last): File "/usr/share/ceph/mgr/prometheus/module.py", line 513, in collect data = self.mod.collect() File "/usr/share/ceph/mgr/mgr_util.py", line 859, in wrapper result = f(*args, **kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1637, in collect self.get_metadata_and_osd_status() File "/usr/share/ceph/mgr/mgr_util.py", line 859, in wrapper result = f(*args, **kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1121, in get_metadata_and_osd_status osd_map = self.get('osd_map') File "/usr/share/ceph/mgr/mgr_module.py", line 1366, in get obj = json.loads(obj) File "/lib/python3.8/json/__init__.py", line 357, in loads return _default_decoder.decode(s) File "/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/lib/python3.8/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 4145 (char 4144)
Actions