Project

General

Profile

Actions

Bug #63992

closed

Prometheus module is crashing during startup

Added by Redouane Kachach Elhichou 4 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Category:
cephadm/monitoring
Target version:
-
% Done:

100%

Source:
Tags:
backport_processed
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

you can find more details on the issue: https://github.com/rook/rook/issues/13527

debug 2024-01-09T18:31:05.181+0000 7ff5cc6ef700  1 mgr load Constructed class from module: progress
debug 2024-01-09T18:31:05.182+0000 7ff5cc6ef700  0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO (2/5)
debug 2024-01-09T18:31:05.192+0000 7ff5cc6ef700 -1 no module 'rook'
debug 2024-01-09T18:31:05.192+0000 7ff5cc6ef700 -1 no module 'rook'
debug 2024-01-09T18:31:05.192+0000 7ff5cc6ef700 -1 mgr load Failed to construct class in 'prometheus'
debug 2024-01-09T18:31:05.193+0000 7ff66ef04700  0 [progress INFO root] Loading...
debug 2024-01-09T18:31:05.201+0000 7ff66ef04700  0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7ff742939080>, <progress.module.GhostEvent object at 0x7ff7429390b8>, <progress.module.GhostEvent object at 0x7ff7429390f0>, <progress.module.GhostEvent object at 0x7ff742939128>, <progress.module.GhostEvent object at 0x7ff742939160>, <progress.module.GhostEvent object at 0x7ff742939198>, <progress.module.GhostEvent object at 0x7ff7429391d0>, <progress.module.GhostEvent object at 0x7ff742939208>, <progress.module.GhostEvent object at 0x7ff742939240>, <progress.module.GhostEvent object at 0x7ff742939278>, <progress.module.GhostEvent object at 0x7ff7429392b0>, <progress.module.GhostEvent object at 0x7ff7429392e8>, <progress.module.GhostEvent object at 0x7ff742939320>, <progress.module.GhostEvent object at 0x7ff742939358>, <progress.module.GhostEvent object at 0x7ff742939390>, <progress.module.GhostEvent object at 0x7ff7429393c8>, <progress.module.GhostEvent object at 0x7ff742939400>, <progress.module.GhostEvent object at 0x7ff742939438>, <progress.module.GhostEvent object at 0x7ff742939470>, <progress.module.GhostEvent object at 0x7ff7429394a8>, <progress.module.GhostEvent object at 0x7ff7429394e0>, <progress.module.GhostEvent object at 0x7ff742939518>, <progress.module.GhostEvent object at 0x7ff742939550>, <progress.module.GhostEvent object at 0x7ff742939588>, <progress.module.GhostEvent object at 0x7ff7429395c0>, <progress.module.GhostEvent object at 0x7ff7429395f8>, <progress.module.GhostEvent object at 0x7ff742939630>, <progress.module.GhostEvent object at 0x7ff742939668>, <progress.module.GhostEvent object at 0x7ff7429396a0>, <progress.module.GhostEvent object at 0x7ff7429396d8>] historic events
debug 2024-01-09T18:31:05.202+0000 7ff66ef04700  0 [progress INFO root] Loaded OSDMap, ready.
debug 2024-01-09T18:31:05.203+0000 7ff67ff06700  0 [pg_autoscaler INFO root] _maybe_adjust
debug 2024-01-09T18:31:05.192+0000 7ff5cc6ef700 -1 mgr load Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
    return mgr.remote(o, meth, *args, **kwargs)
  File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
    args, kwargs)
ImportError: Module not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/prometheus/module.py", line 649, in __init__
    self.modify_instance_id = self.get_orch_status() and self.get_module_option(
  File "/usr/share/ceph/mgr/prometheus/module.py", line 869, in get_orch_status
    return self.available()[0]
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
    completion = self._oremote(method_name, args, kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1661, in _oremote
    f_set = self.get_feature_set()
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1586, in inner
    completion = self._oremote(method_name, args, kwargs)
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1657, in _oremote
    return mgr.remote(o, meth, *args, **kwargs)
  File "/usr/share/ceph/mgr/mgr_module.py", line 2228, in remote
    args, kwargs)
ImportError: Module not found


Related issues 2 (0 open2 closed)

Has duplicate Ceph - Bug #63868: No metrics (wrong mgr opens metrics port)DuplicateRedouane Kachach Elhichou

Actions
Copied to Orchestrator - Backport #64339: reef: Prometheus module is crashing during startupResolvedRedouane Kachach ElhichouActions
Actions #1

Updated by Redouane Kachach Elhichou 4 months ago

  • Assignee changed from avan thakkar to Redouane Kachach Elhichou
Actions #2

Updated by Redouane Kachach Elhichou 4 months ago

  • Has duplicate Bug #63868: No metrics (wrong mgr opens metrics port) added
Actions #3

Updated by Redouane Kachach Elhichou 4 months ago

  • Status changed from New to In Progress
  • Pull request ID set to 55393
Actions #4

Updated by Redouane Kachach Elhichou 4 months ago

  • Target version set to v18.2.2
Actions #5

Updated by Adam King 3 months ago

  • Status changed from In Progress to Pending Backport
  • Backport set to reef
Actions #6

Updated by Backport Bot 3 months ago

  • Copied to Backport #64339: reef: Prometheus module is crashing during startup added
Actions #7

Updated by Backport Bot 3 months ago

  • Tags set to backport_processed
Actions #8

Updated by Redouane Kachach Elhichou 3 months ago

  • Pull request ID changed from 55393 to 55149
Actions #9

Updated by Redouane Kachach Elhichou 3 months ago

  • Status changed from Pending Backport to Resolved
Actions #10

Updated by Ilya Dryomov 2 months ago

  • Target version deleted (v18.2.2)
Actions

Also available in: Atom PDF