Bug #63529
openPython Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3
0%
Description
Our users noticed that Ceph's dashboard is broken in Proxmox Virtual Environment 8. On a more closer investigation, this is apparently caused by an import check in Python modules that use PyO3:
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Traceback (most recent call last):
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from .module import Module, StandbyModule # noqa: F401
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/module.py", line 30, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from .controllers import Router, json_error_page
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from ._api_router import APIRouter
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/controllers/_api_router.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from ._router import Router
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/controllers/_router.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from ._base_controller import BaseController
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from ..services.auth import AuthManager, JwtManager
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/usr/share/ceph/mgr/dashboard/services/auth.py", line 12, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: import jwt
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/jwt/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from .api_jwk import PyJWK, PyJWKSet
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/jwt/api_jwk.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from .algorithms import get_default_algorithms
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/jwt/algorithms.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from .utils import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/jwt/utils.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from cryptography.hazmat._oid import ObjectIdentifier
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: File "/lib/python3/dist-packages/cryptography/hazmat/_oid.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: from cryptography.hazmat.bindings._rust import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: ImportError: PyO3 modules may only be initialized once per interpreter process
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Class not found in module 'dashboard'
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.470+0200 7fecdc91e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard
This error doesn't just appear in PVE, but also in:
- Debian 12 (Bookworm), see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055212
- ArchLinux (AUR), see: https://github.com/bazaah/aur-ceph/issues/20
This is due to the fact that every ceph-mgr
module that uses Python modules that use PyO3 to provide bindings to Rust code will raise an ImportError
if imported more than once under the presence of multiple Python sub-interpreters. This check is present in PyO3 version 0.17.0 and upwards and was introduced in this pull request: https://github.com/PyO3/pyo3/pull/2523
So, for now it seems that this "only" completely breaks the dashboard; however, other ceph-mgr
modules might also be affected in the future, depending on whether they use PyO3 Python modules or not.
This also means that the official distribution of Ceph will be affected sooner or later, either when a newer version of Python's cryptography
or PyO3 is used.
The above is essentially a summary of my findings and posts over at the Proxmox forum. For the curious, you can read up on everything starting from this post over here: https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/post-587100
Updated by Iggy Jackson 4 months ago
`bcrypt` is now also doing this, so I think basically every module is broken at this point.
Updated by Niklas Hambuechen 4 months ago
See also: https://github.com/ceph/ceph/pull/54710
Updated by Matthew Vernon 3 months ago
Since that PR is a partial fix for this, I've put in https://github.com/ceph/ceph/pull/55689 to backport it to reef.