Project

General

Profile

Bug #37472

Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..."

Added by Ricardo Dias over 1 year ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
ceph-mgr
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

The latest version of Cython v0.29 removed the support for python subinterpreters [1, 2]

This creates a serious problem in ceph-mgr because all mgr modules import the rados cython module in their own python subinterpreter.
When running the ceph-mgr daemon in a system with Cython 0.29, all the modules except the first one will raise an exception when loading the module.
The error message is:

2018-10-31 13:52:33.813 7f978235e740 -1 mgr[py] Traceback (most recent call last):
  File "/ceph/src/pybind/mgr/mgr_module.py", line 8, in <module>
    import rados
ImportError: Interpreter change detected - this module can only be loaded into one interpreter per process.

[1]
https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50
[2] http://blog.behnel.de/posts/whats-new-in-cython-029.html


Related issues

Related to mgr - Bug #38407: Funny issues with python sub-interpreters New 02/20/2019
Related to mgr - Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. Resolved 03/17/2019
Copied to mgr - Backport #39593: mimic: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." Resolved

History

#1 Updated by Nathan Cutler over 1 year ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 25328

#2 Updated by Sage Weil over 1 year ago

  • Status changed from Fix Under Review to 12
  • Priority changed from High to Urgent

merged https://github.com/ceph/ceph/pull/25328

I'm not sure we should close this bug though since the packaging change just means we will fail at build time instead of run time; we still need to sort out what to do wrt the cython change.

#3 Updated by Nathan Cutler over 1 year ago

  • Pull request ID deleted (25328)

#4 Updated by Brad Hubbard over 1 year ago

This breaks building on fedora rawhide (fc30) where only 0.29.1-1.fc30 is available.

https://copr-be.cloud.fedoraproject.org/results/badone/ceph-weeklies/fedora-rawhide-x86_64/00836454-ceph/builder-live.log

#5 Updated by Lenz Grimmer over 1 year ago

Unfortunately it seems as if the Cython developers have no intentions to revert this breaking change - we will have to figure out an alternative solution:

https://mail.python.org/pipermail/cython-devel/2018-December/thread.html#5233

#6 Updated by Nathan Cutler over 1 year ago

Brad Hubbard wrote:

This breaks building on fedora rawhide (fc30) where only 0.29.1-1.fc30 is available.

https://copr-be.cloud.fedoraproject.org/results/badone/ceph-weeklies/fedora-rawhide-x86_64/00836454-ceph/builder-live.log

Hi Brad: You're right that the addition of "Cython < 0.29" to the spec file broke the Fedora build, but it was a choice between that and letting distros build a broken mgr (see Ricardo's bug description, above).

In Tumbleweed we had the same problem, and we overcame it by reverting Tumbleweed's Cython package to 0.28.5.

#7 Updated by Tim Serong over 1 year ago

Lenz Grimmer wrote:

Unfortunately it seems as if the Cython developers have no intentions to revert this breaking change - we will have to figure out an alternative solution:

FWIW I'm poking around at this too, to see if I can think of some magic.

https://mail.python.org/pipermail/cython-devel/2018-December/thread.html#5233

Sage added a couple of good replies to that thread already, but somehow they're not yet visible in the list archives :-/

#8 Updated by Brad Hubbard over 1 year ago

Nathan Cutler wrote:

Hi Brad: You're right that the addition of "Cython < 0.29" to the spec file broke the Fedora build, but it was a choice between that and letting distros build a broken mgr (see Ricardo's bug description, above).

I get that. It wasn't my intention to apportion blame or be overly critical of the solution but to simply add another detail to the tracker. Sorry if it came across as accusatory or overly critical.

In Tumbleweed we had the same problem, and we overcame it by reverting Tumbleweed's Cython package to 0.28.5.

Not sure I have the same influence on fedora's direction mate ;)

#10 Updated by Kefu Chai over 1 year ago

  • Status changed from 12 to Resolved

#11 Updated by Mykola Golub over 1 year ago

Shouldn't we consider this fix rather as a workaround and still keep the bug open? The cython developers tell us that what we are doing is not safe in general case and not supported, and we just remove the protection and continued doing this.

#12 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #38407: Funny issues with python sub-interpreters added

#13 Updated by Sebastian Wagner about 1 year ago

  • Duplicated by Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added

#14 Updated by Tim Serong about 1 year ago

  • Status changed from Resolved to Pending Backport
  • Backport set to mimic

#15 Updated by Kefu Chai about 1 year ago

  • Pull request ID set to 25585

Shouldn't we consider this fix rather as a workaround and still keep the bug open? The cython developers tell us that what we are doing is not safe in general case and not supported, and we just remove the protection and continued doing this.

agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?

#16 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #39593: mimic: Cython 0.29 removed support for subinterpreters: raises "ImportError: Interpreter change detected ..." added

#17 Updated by Nathan Cutler about 1 year ago

  • Duplicated by deleted (Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process.)

#18 Updated by Nathan Cutler about 1 year ago

  • Related to Bug #38788: ceph-mgr: ImportError: Interpreter change detected - this module can only be loaded into one interprer per process. added

#19 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

#20 Updated by Niklas Hambuechen about 2 months ago

agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?

Was this done?

Could it be done?

#21 Updated by Brad Hubbard about 2 months ago

Niklas Hambuechen wrote:

agreed. how about using this ticket for tracking the backport, and creating a copy of it for a proper fix in future?

Was this done?

Could it be done?

Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.

Why do you ask?

#22 Updated by Niklas Hambuechen about 2 months ago

Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.
Why do you ask?

The usual reason: Finding this issue after encountering the error message after an upgrade of a Ceph cluster, and seeing it marked as closed with the consensus seemingly being that what was merged was a temporary workaround, and no follow-up or updated decision being linked.

For others encountering this, it seems that the fix has two parts (only one of which is linked from here):

#23 Updated by Brad Hubbard about 2 months ago

Niklas Hambuechen wrote:

Recent discussion resulted in a decision to proceed as we are now, i.e. leave subinterpreters in place.
Why do you ask?

The usual reason: Finding this issue after encountering the error message after an upgrade of a Ceph cluster, and seeing it marked as closed with the consensus seemingly being that what was merged was a temporary workaround, and no follow-up or updated decision being linked.

For others encountering this, it seems that the fix has two parts (only one of which is linked from here):

I'm wondering what made you choose to upgrade to Nautilus < 14.1.0 when 14.2.8 is available?

x.1.z releases are release candidates "(for test clusters, brave users)" and x.0.z are development releases.

#24 Updated by Niklas Hambuechen about 2 months ago

I'm wondering what made you choose to upgrade to Nautilus < 14.1.0

Sorry for being unclear: I did not upgrade Ceph, but an OS upgrade for my cluster brought in the newer Cython version that triggered the error message, but not yet a Ceph new enough to have the workaround in.

#25 Updated by Brad Hubbard about 2 months ago

Niklas Hambuechen wrote:

I'm wondering what made you choose to upgrade to Nautilus < 14.1.0

Sorry for being unclear: I did not upgrade Ceph, but an OS upgrade for my cluster brought in the newer Cython version that triggered the error message, but not yet a Ceph new enough to have the workaround in.

Ah, I see, that's bad luck. This does get discussed and there are people looking for a better direction but it's quite a curly problem to solve. Thanks for the clarification.

Also available in: Atom PDF