Bug #45147: Module 'diskprediction_local' takes forever to load - mgr - Ceph

Actions

Copy link

Bug #45147

closed

Module 'diskprediction_local' takes forever to load

Added by Kefu Chai about 4 years ago. Updated 5 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

diskprediction_local

Target version:

% Done:

100%

Source:

Tags:

backport_processed

Backport:

octopus, pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

34846

Crash signature (v1):

Crash signature (v2):

Description

debian sid, python3.8,2,

sha1: 5e540f1191975fac38733e765312295775a52ad4

ceph-mgr never finishes loading diskprediction_local:

2020-04-20T17:21:04.157+0800 7f2d9860ee40  1 mgr[py] Loading python module 'diskprediction_local'
2020-04-20T17:21:04.201+0800 7f2d9860ee40 10 mgr[py] Computed sys.path '/var/ssd/ceph/src/pybind:/var/ssd/ceph/build/lib/cython_modules/lib.3:/var/ssd/ceph/src/python-common::/usr/lib/python38.zip:/usr/l\
ib/python3.8:/usr/lib/python3.8/lib-dynload:/var/ssd/ceph/src/pybind/mgr:/usr/local/lib/python3.8/dist-packages:/usr/lib/python3/dist-packages:/usr/lib/python3.8/dist-packages'
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:34.435106+0800)
2020-04-20T17:21:04.429+0800 7f2d7d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1
2020-04-20T17:21:05.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:05.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:35.435416+0800)
2020-04-20T17:21:06.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:06.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:36.435629+0800)
2020-04-20T17:21:07.433+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:07.433+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:37.435829+0800)
2020-04-20T17:21:08.429+0800 7f2d7d7fa700 10 monclient: tick
2020-04-20T17:21:08.429+0800 7f2d7d7fa700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2020-04-20T17:20:38.435989+0800)
2020-04-20T17:21:09.429+0800 7f2d7d7fa700 10 monclient: tick

Related issues 6 (1 open — 5 closed)

Actions

Copy link

Updated by Kefu Chai about 4 years ago

Related to Bug #42764: Test failure: test_diskprediction_local (tasks.mgr.test_module_selftest.TestModuleSelftest) added

Actions

Copy link

Updated by Kefu Chai about 4 years ago

i think it's caused by numpy.

see the discussions at

- https://mail.python.org/pipermail/python-dev/2019-January/156095.html and
- https://bugs.python.org/issue10915

seems the short-term fix is to either

- disable diskprediction_local
- run diskprediction_local in a dedicated interpreter instead of in a sub-interpreter

Actions

Copy link

Updated by Tomasz Torcz almost 4 years ago

FYI, the same happens on Fedora 33.

Actions

Copy link

Updated by Kiefer Chang almost 4 years ago

Related to Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versions added

Actions

Copy link

Updated by Kefu Chai almost 4 years ago

Jeff Layton also ran into this issue on fedora 32 with python3.8.

Actions

Copy link

Updated by Josh Durgin almost 4 years ago

Priority changed from Normal to Urgent

Actions

Copy link

Updated by Kefu Chai almost 4 years ago

Status changed from New to Fix Under Review
Assignee set to Kefu Chai
Pull request ID set to 34846

as this issue only impacts python3.8, and neither el8 nor ubuntu bionic ships python3.8. i am not adding any LTS branches to "backport" field of this ticket.

Actions

Copy link