Project

General

Profile

Bug #38788

ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process.

Added by Alexey Shabalin 3 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Category:
ceph-mgr
Target version:
-
Start date:
03/17/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
nautilus, mimic
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Build environment:
- ALT Linux Sisyphus
- python-3.6
- Cython-0.29.6
- ceph-14.1.1

Errors(for all mgr modules) as in https://tracker.ceph.com/issues/37472:

Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.598 7fef63af6d00 -1 mgr[py] Class not found in module 'status'
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.598 7fef63af6d00 -1 mgr[py] Error loading module 'status': (22) Invalid argument
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.641 7fef63af6d00 -1 mgr[py] Module not found: 'mgr_module'
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: 2019-03-17 19:26:41.641 7fef63af6d00 -1 mgr[py] Traceback (most recent call last):
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]:   File "/usr/share/ceph/mgr/mgr_module.py", line 9, in <module>
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]:     import rados
Mar 17 19:26:41 virt-lab02 ceph-mgr[413528]: ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

Please, see build log in attache.

ceph-build.log.xz - log of build ceph-14.1.1 (163 KB) Alexey Shabalin, 03/17/2019 04:58 PM


Related issues

Related to mgr - Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." Pending Backport 11/29/2018
Copied to mgr - Backport #38865: nautilus: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. Resolved
Copied to mgr - Backport #39592: mimic: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. In Progress

History

#1 Updated by Sebastian Wagner 3 months ago

  • Affected Versions v14.0.0 added

#2 Updated by Sebastian Wagner 3 months ago

  • Duplicates Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added

#3 Updated by Tim Serong 3 months ago

Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'") in the build logs, so I'm not yet sure why this isn't working.

The following is possibly unrelated, but there's a bunch of odd cython warnings:

/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx

I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with -DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.

#4 Updated by Tim Serong 3 months ago

OK, I get this stuff too, doing a build of v14.1.1 with cython 0.29.1 on openSUSE Tumbleweed:

/usr/lib64/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/tserong/src/github/SUSE/ceph/src/pybind/rbd/rbd.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

But, if I then run a vstart cluster, mgr works fine. No problem with subinterpreters. I still need to re-test against 0.29.6 for completeness, but a quick look at the diff between 0.29.1 and 0.29.6 doesn't seem to show anything subinterpreter-related.

#5 Updated by Alexey Shabalin 3 months ago

Tim Serong wrote:

Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'") in the build logs, so I'm not yet sure why this isn't working.

The following is possibly unrelated, but there's a bunch of odd cython warnings:

/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx

I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."

I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with -DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.

Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.

#6 Updated by Tim Serong 3 months ago

Alexey Shabalin wrote:

I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."

[...]

Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.

Thanks for letting me know. I think the python2 thing is unrelated, so let's ignore that for now.

How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?

#7 Updated by Tim Serong 3 months ago

  • Assignee set to Tim Serong

#8 Updated by Alexey Shabalin 3 months ago

Tim Serong wrote:

How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?

Yes, you can try Starterkits/builder, but i think builder pre-configured for use with stable branch/p8 (apt repo).
May be easy use docker image:
1) docker run -it --privileged alt:sisyphus bash
2) apt-get update
3) apt-get -y dist-upgrade
4) apt-get install -y su hasher gear
5) useradd -m build
6) hasher-useradd build
7) su - build
8) git clone git://git.altlinux.org/people/shaba/packages/ceph.git
9) cd ceph

10) gear-hsh
or
10.1) gear ~/tmp/ceph-14.1.1-alt1.gear
10.2) hsh -v ~/hasher ~/tmp/ceph-14.1.1-alt1.gear

#9 Updated by Tim Serong 3 months ago

Thanks, I have at least reproduced the problem here with the alt-linux build. The "easy" test of whether it's working or not is strings /usr/lib64/python3/site-packages/rados.cpython-36m.so|grep 'Interpreter change'. That should print nothing if the check has been disabled (i.e. the __Pyx_check_single_interpreter function won't exist). If it prints "Interpreter change detected - this module can only be loaded into one interpreter per process", we know it's broken.

Looking closer at the build, I found something weird. Check the build logs -- there's two sections in the build output for each extension module, one starting with "[1/1] Cythonizing rados.pyx", which correctly includes the -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 flags when compiling, and a second section which starts with "cythoning rados.pyx to /usr/src/RPM/BUILD/ceph-14.1.1/build/src/pybind/rados3/pyrex/rados.c" This second section does not include those defines. BOTH sections output the file /usr/src/RPM/BUILD/ceph-14.1.1/build/lib/cython_modules/lib.3/rados.cpython-36m.so, so the second one is overwriting the first. That explains the problem, but does not explain why I don't have the problem when doing a regular build out of the source tree. Also, WTF? Why is it building rados.pyx (and the other python extension modules) twice, in slightly different ways?

#10 Updated by Tim Serong 3 months ago

The first (correct) build of the extension libraries happens during the regular build (`make`). The second one, which overwrites the first, happens during `make install`. So of course that's why I didn't see it when building by hand out of the source tree then running vstart clusters -- `make install` never happened in that case! So, we either need to get rid of that presumably redundant second compile of the extension modules during `make install`, or fix it so it also picks up the flags that will disable the subinterpreter check.

#11 Updated by Tim Serong 3 months ago

  • Status changed from New to In Progress
  • Priority changed from Normal to Urgent

#12 Updated by Tim Serong 3 months ago

  • Status changed from In Progress to Need Review
  • Backport set to nautilus
  • Pull request ID set to 27067

Alexy, can you please try a build with https://github.com/ceph/ceph/pull/27067 applied?

#13 Updated by Alexey Shabalin 3 months ago

Tim Serong wrote:

Alexy, can you please try a build with https://github.com/ceph/ceph/pull/27067 applied?

Yes, this patch fixed error.
Thanks you!

#14 Updated by Kefu Chai 3 months ago

  • Status changed from Need Review to Pending Backport

#15 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #38865: nautilus: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added

#16 Updated by Nathan Cutler 3 months ago

  • Status changed from Pending Backport to Resolved

#17 Updated by Tim Serong about 2 months ago

  • Status changed from Resolved to Pending Backport
  • Backport changed from nautilus to nautilus, mimic

#18 Updated by Nathan Cutler about 2 months ago

  • Copied to Backport #39592: mimic: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added

#19 Updated by Nathan Cutler about 2 months ago

  • Duplicates deleted (Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ...")

#20 Updated by Nathan Cutler about 2 months ago

  • Related to Bug #37472: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added

Also available in: Atom PDF