Project

General

Profile

Bug #51240

mgr module fails in focal, due to ceph-mgr-rook module

Added by Deepika Upadhyay almost 3 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
ceph-mgr
Target version:
-
% Done:

100%

Source:
Tags:
backport_processed
Backport:
pacific, octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

The 'devicehealth' plugins' dependency on rook (package ceph-mgr-rook) code causes a cluster to not boot, after upgrade to mimic v15.2.1 on Debian Buster.
Seems apparently to be due to interaction between Rook code and python3-numpy (version 1:1.16.2-1), and not unique to Rook ( https://github.com/numpy/numpy/issues/14384 ).
Fix not available upstream, seems "WON'T IMPLEMENT", so fix required in Rook.

May 17 21:01:42 davinci ceph-mgr[82022]: 2020-05-17T21:01:42.587+0200 7f262360ff40 -1 mgr[py] Module not found: 'rook'
May 17 21:01:42 davinci ceph-mgr[82022]: 2020-05-17T21:01:42.587+0200 7f262360ff40 -1 mgr[py] Traceback (most recent call last):
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from .module import RookOrchestrator
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/share/ceph/mgr/rook/module.py", line 16, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from kubernetes import client, config
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/kubernetes/__init__.py", line 22, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     import kubernetes.stream
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/kubernetes/stream/__init__.py", line 15, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from .stream import stream
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/kubernetes/stream/stream.py", line 13, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from . import ws_client
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/kubernetes/stream/ws_client.py", line 19, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from websocket import WebSocket, ABNF, enableTrace
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/websocket/__init__.py", line 22, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from ._abnf import *
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/websocket/_abnf.py", line 34, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     import numpy
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from . import core
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from . import multiarray
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     from . import overrides
May 17 21:01:42 davinci ceph-mgr[82022]:   File "/usr/lib/python3/dist-packages/numpy/core/overrides.py", line 46, in <module>
May 17 21:01:42 davinci ceph-mgr[82022]:     """)
May 17 21:01:42 davinci ceph-mgr[82022]: RuntimeError: implement_array_function method already has a docstring
May 17 21:01:42 davinci ceph-mgr[82022]: 2020-05-17T21:01:42.591+0200 7f262360ff40 -1 mgr[py] Class not found in module 'rook'
May 17 21:01:42 davinci ceph-mgr[82022]: 2020-05-17T21:01:42.591+0200 7f262360ff40 -1 mgr[py] Error loading module 'rook': (2) No such file or directory
May 17 21:01:43 davinci ceph-mgr[82022]: 2020-05-17T21:01:43.099+0200 7f262360ff40 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook
May 17 21:01:46 davinci ceph-mgr[82022]: 2020-05-17T21:01:46.211+0200 7f260ae73700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.davinci.lund.millnert.se:
May 17 21:01:46 davinci ceph-mgr[82022]: 2020-05-17T21:01:46.211+0200 7f260ae73700 -1 devicehealth.serve:
May 17 21:01:46 davinci ceph-mgr[82022]: 2020-05-17T21:01:46.211+0200 7f260ae73700 -1 Traceback (most recent call last):
May 17 21:01:46 davinci ceph-mgr[82022]:   File "/usr/share/ceph/mgr/devicehealth/module.py", line 260, in serve
May 17 21:01:46 davinci ceph-mgr[82022]:     self.scrape_all()
May 17 21:01:46 davinci ceph-mgr[82022]:   File "/usr/share/ceph/mgr/devicehealth/module.py", line 327, in scrape_all
May 17 21:01:46 davinci ceph-mgr[82022]:     ioctx = self.open_connection()
May 17 21:01:46 davinci ceph-mgr[82022]:   File "/usr/share/ceph/mgr/devicehealth/module.py", line 297, in open_connection
May 17 21:01:46 davinci ceph-mgr[82022]:     assert r == 0
May 17 21:01:46 davinci ceph-mgr[82022]: AssertionError


Related issues

Copied from mgr - Bug #45574: subinterpreters: ceph/mgr/rook RuntimeError on import of RookOrchestrator - ceph cluster does not start New
Copied to mgr - Backport #51241: pacific: mgr module fails in focal, due to ceph-mgr-rook module Resolved
Copied to mgr - Backport #51242: octopus: mgr module fails in focal, due to ceph-mgr-rook module Rejected

History

#1 Updated by Deepika Upadhyay almost 3 years ago

  • Copied from Bug #45574: subinterpreters: ceph/mgr/rook RuntimeError on import of RookOrchestrator - ceph cluster does not start added

#2 Updated by Deepika Upadhyay almost 3 years ago

  • Status changed from New to Pending Backport

#3 Updated by Deepika Upadhyay almost 3 years ago

  • Copied to Backport #51241: pacific: mgr module fails in focal, due to ceph-mgr-rook module added

#4 Updated by Deepika Upadhyay almost 3 years ago

  • Copied to Backport #51242: octopus: mgr module fails in focal, due to ceph-mgr-rook module added

#5 Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed

#6 Updated by Konstantin Shalygin 11 months ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF