Bug #37472
closedCython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..."
Added by Ricardo Dias over 5 years ago. Updated about 4 years ago.
0%
Description
The latest version of Cython v0.29 removed the support for python subinterpreters [1, 2]
This creates a serious problem in ceph-mgr because all mgr modules import the rados cython module in their own python subinterpreter.
When running the ceph-mgr daemon in a system with Cython 0.29, all the modules except the first one will raise an exception when loading the module.
The error message is:
2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Traceback (most recent call last):
File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.
[1]
https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50
[2] http://blog.behnel.de/posts/whats-new-in-cython-029.html
Updated by Nathan Cutler over 5 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 25328
Updated by Sage Weil over 5 years ago
- Status changed from Fix Under Review to 12
- Priority changed from High to Urgent
merged https://github.com/ceph/ceph/pull/25328
I'm not sure we should close this bug though since the packaging change just means we will fail at build time instead of run time; we still need to sort out what to do wrt the cython change.
Updated by Brad Hubbard over 5 years ago
This breaks building on fedora rawhide (fc30) where only 0.29.1-1.fc30 is available.
Updated by Lenz Grimmer over 5 years ago
Unfortunately it seems as if the Cython developers have no intentions to revert this breaking change - we will have to figure out an alternative solution:
https://mail.python.org/pipermail/cython-devel/2018-December/thread.html#5233
Updated by Nathan Cutler over 5 years ago
Brad Hubbard wrote:
This breaks building on fedora rawhide (fc30) where only 0.29.1-1.fc30 is available.
Hi Brad: You're right that the addition of "Cython < 0.29" to the spec file broke the Fedora build, but it was a choice between that and letting distros build a broken mgr (see Ricardo's bug description, above).
In Tumbleweed we had the same problem, and we overcame it by reverting Tumbleweed's Cython package to 0.28.5.
Updated by Tim Serong over 5 years ago
Lenz Grimmer wrote:
Unfortunately it seems as if the Cython developers have no intentions to revert this breaking change - we will have to figure out an alternative solution:
FWIW I'm poking around at this too, to see if I can think of some magic.
https://mail.python.org/pipermail/cython-devel/2018-December/thread.html#5233
Sage added a couple of good replies to that thread already, but somehow they're not yet visible in the list archives :-/
Updated by Brad Hubbard over 5 years ago
Nathan Cutler wrote:
Hi Brad: You're right that the addition of "Cython < 0.29" to the spec file broke the Fedora build, but it was a choice between that and letting distros build a broken mgr (see Ricardo's bug description, above).
I get that. It wasn't my intention to apportion blame or be overly critical of the solution but to simply add another detail to the tracker. Sorry if it came across as accusatory or overly critical.
In Tumbleweed we had the same problem, and we overcame it by reverting Tumbleweed's Cython package to 0.28.5.
Not sure I have the same influence on fedora's direction mate ;)
Updated by Tim Serong over 5 years ago
I have an evil fix: https://github.com/ceph/ceph/pull/25585
Updated by Mykola Golub over 5 years ago
Shouldn't we consider this fix rather as a workaround and still keep the bug open? The cython developers tell us that what we are doing is not safe in general case and not supported, and we just remove the protection and continued doing this.
Updated by Sebastian Wagner about 5 years ago
- Related to Bug #38407: Funny issues with python sub-interpreters added
Updated by Sebastian Wagner about 5 years ago
- Has duplicate Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added
Updated by Tim Serong almost 5 years ago
- Status changed from Resolved to Pending Backport
- Backport set to mimic
Seen in v13.2.4 on Ubuntu Disco (https://github.com/ceph/ceph/pull/25585#issuecomment-489016057)
Updated by Kefu Chai almost 5 years ago
- Pull request ID set to 25585
Shouldn't we consider this fix rather as a workaround and still keep the bug open? The cython developers tell us that what we are doing is not safe in general case and not supported, and we just remove the protection and continued doing this.
agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?
Updated by Nathan Cutler almost 5 years ago
- Copied to Backport #39593: mimic: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added
Updated by Nathan Cutler almost 5 years ago
- Has duplicate deleted (Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process.)
Updated by Nathan Cutler almost 5 years ago
- Related to Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added
Updated by Nathan Cutler almost 5 years ago
- Status changed from Pending Backport to Resolved
Updated by Niklas Hambuechen about 4 years ago
agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?
Was this done?
Could it be done?
Updated by Brad Hubbard about 4 years ago
Niklas Hambuechen wrote:
agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?
Was this done?
Could it be done?
Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.
Why do you ask?
Updated by Niklas Hambuechen about 4 years ago
Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.
Why do you ask?
The usual reason: Finding this issue after encountering the error message after an upgrade of a Ceph cluster, and seeing it marked as closed with the consensus seemingly being that what was merged was a temporary workaround, and no follow-up or updated decision being linked.
For others encountering this, it seems that the fix has two parts (only one of which is linked from here):
Updated by Brad Hubbard about 4 years ago
Niklas Hambuechen wrote:
Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.
Why do you ask?The usual reason: Finding this issue after encountering the error message after an upgrade of a Ceph cluster, and seeing it marked as closed with the consensus seemingly being that what was merged was a temporary workaround, and no follow-up or updated decision being linked.
For others encountering this, it seems that the fix has two parts (only one of which is linked from here):
I'm wondering what made you choose to upgrade to Nautilus < 14.1.0 when 14.2.8 is available?
x.1.z releases are release candidates "(for test clusters, brave users)" and x.0.z are development releases.
Updated by Niklas Hambuechen about 4 years ago
I'm wondering what made you choose to upgrade to Nautilus < 14.1.0
Sorry for being unclear: I did not upgrade Ceph, but an OS upgrade for my cluster brought in the newer Cython version that triggered the error message, but not yet a Ceph new enough to have the workaround in.
Updated by Brad Hubbard about 4 years ago
Niklas Hambuechen wrote:
I'm wondering what made you choose to upgrade to Nautilus < 14.1.0
Sorry for being unclear: I did not upgrade Ceph, but an OS upgrade for my cluster brought in the newer Cython version that triggered the error message, but not yet a Ceph new enough to have the workaround in.
Ah, I see, that's bad luck. This does get discussed and there are people looking for a better direction but it's quite a curly problem to solve. Thanks for the clarification.