Bug #38788
closedceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process.
Added by Alexey Shabalin about 5 years ago. Updated almost 5 years ago.
0%
Description
Build environment:
- ALT Linux Sisyphus
- python-3.6
- Cython-0.29.6
- ceph-14.1.1
Errors(for all mgr modules) as in https://tracker.ceph.com/issues/37472:
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.598 7fef63af6d00 -1 mgr[py] Class not found in module 'status' Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.598 7fef63af6d00 -1 mgr[py] Error loading module 'status': (22) Invalid argument Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.641 7fef63af6d00 -1 mgr[py] Module not found: 'mgr_module' Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.641 7fef63af6d00 -1 mgr[py] Traceback (most recent call last): Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: File "/usr/share/ceph/mgr/mgr_module.py", line 9, in <module> Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: import rados Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.
Please, see build log in attache.
Files
ceph-build.log.xz (163 KB) ceph-build.log.xz | log of build ceph-14.1.1 | Alexey Shabalin, 03/17/2019 04:58 PM |
Updated by Sebastian Wagner about 5 years ago
- Is duplicate of Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added
Updated by Tim Serong about 5 years ago
Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'"
) in the build logs, so I'm not yet sure why this isn't working.
The following is possibly unrelated, but there's a bunch of odd cython warnings:
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx /usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx
I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with -DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF
. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.
Updated by Tim Serong about 5 years ago
OK, I get this stuff too, doing a build of v14.1.1 with cython 0.29.1 on openSUSE Tumbleweed:
/usr/lib64/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/tserong/src/github/SUSE/ceph/src/pybind/rbd/rbd.pyx tree = Parsing.p_module(s, pxd, full_module_name)
But, if I then run a vstart cluster, mgr works fine. No problem with subinterpreters. I still need to re-test against 0.29.6 for completeness, but a quick look at the diff between 0.29.1 and 0.29.6 doesn't seem to show anything subinterpreter-related.
Updated by Alexey Shabalin about 5 years ago
Tim Serong wrote:
Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (
-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'"
) in the build logs, so I'm not yet sure why this isn't working.The following is possibly unrelated, but there's a bunch of odd cython warnings:
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx
I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."
I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with
-DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF
. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.
Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.
Updated by Tim Serong about 5 years ago
Alexey Shabalin wrote:
I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."
[...]
Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.
Thanks for letting me know. I think the python2 thing is unrelated, so let's ignore that for now.
How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?
Updated by Alexey Shabalin about 5 years ago
Tim Serong wrote:
How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?
Yes, you can try Starterkits/builder, but i think builder pre-configured for use with stable branch/p8 (apt repo).
May be easy use docker image:
1) docker run -it --privileged alt:sisyphus bash
2) apt-get update
3) apt-get -y dist-upgrade
4) apt-get install -y su hasher gear
5) useradd -m build
6) hasher-useradd build
7) su - build
8) git clone git://git.altlinux.org/people/shaba/packages/ceph.git
9) cd ceph
10) gear-hsh
or
10.1) gear ~/tmp/ceph-14.1.1-alt1.gear
10.2) hsh -v ~/hasher ~/tmp/ceph-14.1.1-alt1.gear
Updated by Tim Serong about 5 years ago
Thanks, I have at least reproduced the problem here with the alt-linux build. The "easy" test of whether it's working or not is strings /usr/lib64/python3/site-packages/rados.cpython-36m.so|grep 'Interpreter change'
. That should print nothing if the check has been disabled (i.e. the __Pyx_check_single_interpreter function won't exist). If it prints "Interpreter change detected - this module can only be loaded into one interpreter per process", we know it's broken.
Looking closer at the build, I found something weird. Check the build logs -- there's two sections in the build output for each extension module, one starting with "[1/1] Cythonizing rados.pyx", which correctly includes the -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0
flags when compiling, and a second section which starts with "cythoning rados.pyx to /usr/src/RPM/BUILD/ceph-14.1.1/build/src/pybind/rados3/pyrex/rados.c" This second section does not include those defines. BOTH sections output the file /usr/src/RPM/BUILD/ceph-14.1.1/build/lib/cython_modules/lib.3/rados.cpython-36m.so, so the second one is overwriting the first. That explains the problem, but does not explain why I don't have the problem when doing a regular build out of the source tree. Also, WTF? Why is it building rados.pyx (and the other python extension modules) twice, in slightly different ways?
Updated by Tim Serong about 5 years ago
The first (correct) build of the extension libraries happens during the regular build (`make`). The second one, which overwrites the first, happens during `make install`. So of course that's why I didn't see it when building by hand out of the source tree then running vstart clusters -- `make install` never happened in that case! So, we either need to get rid of that presumably redundant second compile of the extension modules during `make install`, or fix it so it also picks up the flags that will disable the subinterpreter check.
Updated by Tim Serong about 5 years ago
- Status changed from New to In Progress
- Priority changed from Normal to Urgent
Updated by Tim Serong about 5 years ago
- Status changed from In Progress to Fix Under Review
- Backport set to nautilus
- Pull request ID set to 27067
Alexy, can you please try a build with https://github.com/ceph/ceph/pull/27067 applied?
Updated by Alexey Shabalin about 5 years ago
Tim Serong wrote:
Alexy, can you please try a build with https://github.com/ceph/ceph/pull/27067 applied?
Yes, this patch fixed error.
Thanks you!
Updated by Kefu Chai about 5 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38865: nautilus: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added
Updated by Nathan Cutler about 5 years ago
- Status changed from Pending Backport to Resolved
Updated by Tim Serong almost 5 years ago
- Status changed from Resolved to Pending Backport
- Backport changed from nautilus to nautilus, mimic
Seen in v13.2.4 on Ubuntu Disco (https://github.com/ceph/ceph/pull/25585#issuecomment-489016057)
Updated by Nathan Cutler almost 5 years ago
- Copied to Backport #39592: mimic: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added
Updated by Nathan Cutler almost 5 years ago
- Is duplicate of deleted (Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ...")
Updated by Nathan Cutler almost 5 years ago
- Related to Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added
Updated by Nathan Cutler almost 5 years ago
- Status changed from Pending Backport to Resolved