Bug #63529: Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3 - mgr - Ceph

Actions

Copy link

Bug #63529

open

Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3

Added by Max Carrara 6 months ago. Updated 2 months ago.

Status:

New

Priority:

Normal

Assignee:

Category:

ceph-mgr

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v18.2.0

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Our users noticed that Ceph's dashboard is broken in Proxmox Virtual Environment 8. On a more closer investigation, this is apparently caused by an import check in Python modules that use PyO3:

Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Traceback (most recent call last):
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .module import Module, StandbyModule  # noqa: F401
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 30, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .controllers import Router, json_error_page
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._api_router import APIRouter
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_api_router.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._router import Router
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_router.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._base_controller import BaseController
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ..services.auth import AuthManager, JwtManager
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/services/auth.py", line 12, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     import jwt
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .api_jwk import PyJWK, PyJWKSet
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/api_jwk.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .algorithms import get_default_algorithms
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/algorithms.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .utils import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/utils.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat._oid import ObjectIdentifier
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/_oid.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.bindings._rust import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: ImportError: PyO3 modules may only be initialized once per interpreter process
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Class not found in module 'dashboard'
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.470+0200 7fecdc91e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard

This error doesn't just appear in PVE, but also in:

Debian 12 (Bookworm), see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055212
ArchLinux (AUR), see: https://github.com/bazaah/aur-ceph/issues/20

This is due to the fact that every ceph-mgr module that uses Python modules that use PyO3 to provide bindings to Rust code will raise an ImportError if imported more than once under the presence of multiple Python sub-interpreters. This check is present in PyO3 version 0.17.0 and upwards and was introduced in this pull request: https://github.com/PyO3/pyo3/pull/2523

So, for now it seems that this "only" completely breaks the dashboard; however, other ceph-mgr modules might also be affected in the future, depending on whether they use PyO3 Python modules or not.

This also means that the official distribution of Ceph will be affected sooner or later, either when a newer version of Python's cryptography or PyO3 is used.

The above is essentially a summary of my findings and posts over at the Proxmox forum. For the curious, you can read up on everything starting from this post over here: https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/post-587100