Ceph » mgr

Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'") in the build logs, so I'm not yet sure why this isn't working.

The following is possibly unrelated, but there's a bunch of odd cython warnings:

/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rados/rados.pxd
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rbd/rbd.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx
/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/rgw/rgw.pyx

I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with -DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.

OK, I get this stuff too, doing a build of v14.1.1 with cython 0.29.1 on openSUSE Tumbleweed:

/usr/lib64/python3.6/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /home/tserong/src/github/SUSE/ceph/src/pybind/rbd/rbd.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

But, if I then run a vstart cluster, mgr works fine. No problem with subinterpreters. I still need to re-test against 0.29.6 for completeness, but a quick look at the diff between 0.29.1 and 0.29.6 doesn't seem to show anything subinterpreter-related.

Tim Serong wrote:

Well, the fix for #37472 is definitely in this build, as can be confirmed by the presence of (-D'void0=dead_function(void)' -D'__Pyx_check_single_interpreter(ARG)=ARG ## 0'") in the build logs, so I'm not yet sure why this isn't working.

The following is possibly unrelated, but there's a bunch of odd cython warnings:

/usr/lib64/python3/site-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /usr/src/RPM/BUILD/ceph-14.1.1/src/pybind/cephfs/cephfs.pyx

I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."

I don't know exactly what those mean, but it's slightly disturbing to see a mention of Py2 when cmake was run with -DWITH_PYTHON3=ON -DWITH_PYTHON2=OFF. Also, if I'm reading the logs correctly, it looks like python-base-2.7 and libpython-2.7.15 are installed in the build chroot, as well as python3.

Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.

Alexey Shabalin wrote:

I was try apply patch(https://github.com/ceph/ceph/pull/26607) for set 'language_level' for python3, but whatever i have error "ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process."

[...]

Yes, python2 and python3 exist in build chroot together, because we are don't finished migrate to python3 as main in repo. Python2 included in base system now.

Thanks for letting me know. I think the python2 thing is unrelated, so let's ignore that for now.

How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?

Tim Serong wrote:

How can I reproduce your build myself? Looks like I need to use https://en.altlinux.org/Starterkits/builder. Once I have that, what commands are you using to do the build? What git repo are you getting ceph from?

Yes, you can try Starterkits/builder, but i think builder pre-configured for use with stable branch/p8 (apt repo).
May be easy use docker image:
1) docker run -it --privileged alt:sisyphus bash
2) apt-get update
3) apt-get -y dist-upgrade
4) apt-get install -y su hasher gear
5) useradd -m build
6) hasher-useradd build
7) su - build
8) git clone git://git.altlinux.org/people/shaba/packages/ceph.git
9) cd ceph

10) gear-hsh
or
10.1) gear ~/tmp/ceph-14.1.1-alt1.gear
10.2) hsh -v ~/hasher ~/tmp/ceph-14.1.1-alt1.gear

Thanks, I have at least reproduced the problem here with the alt-linux build. The "easy" test of whether it's working or not is strings /usr/lib64/python3/site-packages/rados.cpython-36m.so|grep 'Interpreter change'. That should print nothing if the check has been disabled (i.e. the __Pyx_check_single_interpreter function won't exist). If it prints "Interpreter change detected - this module can only be loaded into one interpreter per process", we know it's broken.

Looking closer at the build, I found something weird. Check the build logs -- there's two sections in the build output for each extension module, one starting with "[1/1] Cythonizing rados.pyx", which correctly includes the -Dvoid0=dead_function(void) -D__Pyx_check_single_interpreter(ARG)=ARG ## 0 flags when compiling, and a second section which starts with "cythoning rados.pyx to /usr/src/RPM/BUILD/ceph-14.1.1/build/src/pybind/rados3/pyrex/rados.c" This second section does not include those defines. BOTH sections output the file /usr/src/RPM/BUILD/ceph-14.1.1/build/lib/cython_modules/lib.3/rados.cpython-36m.so, so the second one is overwriting the first. That explains the problem, but does not explain why I don't have the problem when doing a regular build out of the source tree. Also, WTF? Why is it building rados.pyx (and the other python extension modules) twice, in slightly different ways?

The first (correct) build of the extension libraries happens during the regular build (`make`). The second one, which overwrites the first, happens during `make install`. So of course that's why I didn't see it when building by hand out of the source tree then running vstart clusters -- `make install` never happened in that case! So, we either need to get rid of that presumably redundant second compile of the extension modules during `make install`, or fix it so it also picks up the flags that will disable the subinterpreter check.

Tim Serong wrote:

Alexy, can you please try a build with https://github.com/ceph/ceph/pull/27067 applied?

Yes, this patch fixed error.
Thanks you!