Project

General

Profile

Bug #22612

ceph-mgr's prometheus module crashes when osd is out

Added by Tobias Florek over 1 year ago. Updated over 1 year ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
01/08/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Using ceph-mgr-12.2.2-0.el7.x86_64 when an osd is out (`ceph osd out <id>`) and one curls the metrcis, I get the folowing backtrace.

[08/Jan/2018:13:16:21] HTTP Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in call
self.body = self.oldhandler(args, **kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in call
return self.callable(*self.args, **self.kwargs)
File "/usr/lib64/ceph/mgr/prometheus/module.py", line 386, in metrics
metrics = global_instance().collect()
File "/usr/lib64/ceph/mgr/prometheus/module.py", line 324, in collect
self.get_pg_status()
File "/usr/lib64/ceph/mgr/prometheus/module.py", line 266, in get_pg_status
self.metrics[path].set(value)
KeyError: 'pg_recovery_wait'
[08/Jan/2018:13:16:21] HTTP
Request Headers:
HOST: 88.99.93.16:9283
Remote-Addr: ::ffff:88.99.140.79
ACCEPT: */

USER-AGENT: curl/7.29.0


Related issues

Duplicates mgr - Bug #22116: prometheus module 500 if 'deep' in pg states Resolved 11/13/2017

History

#1 Updated by John Spray over 1 year ago

  • Status changed from New to Duplicate

Should be fixed by:

commit 67a1b66b06bf74274c2cf1d4b5c900f2c56a990d
Author: Jan Fajerski <jfajerski@suse.com>
Date:   Mon Nov 13 14:42:56 2017 +0100

    pybind/mgr/prometheus: don't crash when encountering an unknown PG state

    Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Which is enqueued for backport at https://github.com/ceph/ceph/pull/19138

#2 Updated by Nathan Cutler over 1 year ago

  • Duplicates Bug #22116: prometheus module 500 if 'deep' in pg states added

Also available in: Atom PDF