Ceph » mgr

Starting Python 3.12, CPython supports instantiating subinterpreters with their own GIL.

This would solve performance issues and allow ceph-mgr modules to fully utilize multiple cores (so far, ceph-mgr was performance is limited to a single core).

Additionally, this improves mgr module isolation, which could (perhaps) allow to restart only specific mgr-modules, instead of respawning the whole ceph-mgr everytime a module is enabled or disabled.

In order for this to work, the Py_NewInterpreter() has to be replaced with:

PyInterpreterConfig config = {
    .check_multi_interp_extensions = 1,
    .gil = PyInterpreterConfig_OWN_GIL,
};
PyThreadState *tstate = NULL;
PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);

• After this change, issues might start happening with data-sharing across sub-interpreters, like the ceph-mgr remote() method. This could still be an issue without per-subinterpreter GIL, but with independent GILs issues would be more prone to appear.
• No major LTS distro release brings Python 3.12 (it was released by Oct 3rd, 2023). Whenever, CentOS/Ubuntu/Debian start adopting it, we'll need to provide updated packages for the Python dependencies (this could be partly solved by embedding Python deps).

• Funny issues with python sub-interpreters
• subinterpreters: ceph/mgr/rook RuntimeError on import of RookOrchestrator - ceph cluster does not start

• https://docs.python.org/3.12/whatsnew/3.12.html#pep-684-a-per-interpreter-gil
• https://peps.python.org/pep-0684/
• https://github.com/python/cpython/pull/104210