Project

General

Profile

Bug #22474

prometheus plugin breaks if an osd is out

Added by Ilya Margolin almost 2 years ago. Updated almost 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
Start date:
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

I've taken out an osd (ceph osd out osd.10203) and requested http://mgr:9283/metrics while having degraded PGs, which resulted in 500 Internal Server Error with the following stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 217, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 61, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/ceph/mgr/prometheus/module.py", line 386, in metrics
    metrics = global_instance().collect()
  File "/usr/lib/ceph/mgr/prometheus/module.py", line 324, in collect
    self.get_pg_status()
  File "/usr/lib/ceph/mgr/prometheus/module.py", line 266, in get_pg_status
    self.metrics[path].set(value)
KeyError: 'pg_backfill_wait'

Related issues

Duplicates mgr - Bug #22116: prometheus module 500 if 'deep' in pg states Resolved 11/13/2017

History

#1 Updated by Ilya Margolin almost 2 years ago

Sorry, that's a duplicate of http://tracker.ceph.com/issues/22337

#2 Updated by John Spray almost 2 years ago

  • Duplicates Bug #22116: prometheus module 500 if 'deep' in pg states added

#3 Updated by John Spray almost 2 years ago

  • Status changed from New to Duplicate

Also available in: Atom PDF