Project

General

Profile

Bug #45147

Module 'diskprediction_local' takes forever to load

Added by Kefu Chai 4 months ago. Updated 3 months ago.

Status:
New
Priority:
Urgent
Assignee:
-
Category:
diskprediction_local
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

debian sid, python3.8,2,

sha1: 5e540f1191975fac38733e765312295775a52ad4

ceph-mgr never finishes loading diskprediction_local:

2020-04-20T17:21:04.157+0800 7f2d9860ee40  1 mgr[py] Loading python module 'diskprediction_local'
2020-04-20T17:21:04.201+0800 7f2d9860ee40 10 mgr[py] Computed sys.path '/var/ssd/ceph/src/pybind:/var/ssd/ceph/build/lib/cython_modules/lib.3:/var/ssd/ceph/src/python-common::/usr/lib/python38.zip:/usr/l\
ib/python3.8:/usr/lib/python3.8/lib-dynload:/var/ssd/ceph/src/pybind/mgr:/usr/local/lib/python3.8/dist-packages:/usr/lib/python3/dist-packages:/usr/lib/python3.8/dist-packages'
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:34.435106+0800)
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1
2020-04-20T17:21:05.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:05.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:35.435416+0800)
2020-04-20T17:21:06.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:06.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:36.435629+0800)
2020-04-20T17:21:07.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:07.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:37.435829+0800)
2020-04-20T17:21:08.429+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:08.429+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:38.435989+0800)
2020-04-20T17:21:09.429+0800 7f2d7d7fa700 10 monclient: tick

Related issues

Related to mgr - Bug #42764: Test failure: test_diskprediction_local (tasks.mgr.test_module_selftest.TestModuleSelftest) Resolved
Related to mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versions New

History

#1 Updated by Kefu Chai 4 months ago

  • Related to Bug #42764: Test failure: test_diskprediction_local (tasks.mgr.test_module_selftest.TestModuleSelftest) added

#2 Updated by Kefu Chai 4 months ago

i think it's caused by numpy.

see the discussions at

- https://mail.python.org/pipermail/python-dev/2019-January/156095.html and
- https://bugs.python.org/issue10915

seems the short-term fix is to either

- disable diskprediction_local
- run diskprediction_local in a dedicated interpreter instead of in a sub-interpreter

#3 Updated by Tomasz Torcz 4 months ago

FYI, the same happens on Fedora 33.

#4 Updated by Kiefer Chang 4 months ago

  • Related to Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versions added

#5 Updated by Kefu Chai 4 months ago

Jeff Layton also ran into this issue on fedora 32 with python3.8.

#6 Updated by Josh Durgin 4 months ago

  • Priority changed from Normal to Urgent

#7 Updated by Kefu Chai 4 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Kefu Chai
  • Pull request ID set to 34846

as this issue only impacts python3.8, and neither el8 nor ubuntu bionic ships python3.8. i am not adding any LTS branches to "backport" field of this ticket.

#8 Updated by Kefu Chai 4 months ago

  • Status changed from Fix Under Review to New

the workaround fix was merged. let's keep this issue open until we have a proper fix.

#9 Updated by Kefu Chai 4 months ago

  • Assignee deleted (Kefu Chai)
  • Priority changed from Urgent to High

#10 Updated by Kefu Chai 3 months ago

  • Priority changed from High to Urgent

https://github.com/ceph/ceph/pull/34894 fails the tests on ubuntu focal.

we should address the test failures on ubuntu/focal

Also available in: Atom PDF