https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2020-01-02T17:10:30ZCeph mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1549092020-01-02T17:10:30ZVolker Theile
<ul><li><strong>File</strong> <a href="/attachments/download/4651/mgr.x.log">mgr.x.log</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/4651/mgr.x.log">View</a> added</li></ul> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1549102020-01-02T17:17:47ZVolker Theile
<ul><li><strong>Assignee</strong> set to <i>Sage Weil</i></li></ul><p>@Sage I assigned the issue to you because you already worked on that issue some time ago: <a class="external" href="https://github.com/ceph/ceph/pull/30217/commits/81280455112b1c8f4495ef63b9a5e236e5c65465">https://github.com/ceph/ceph/pull/30217/commits/81280455112b1c8f4495ef63b9a5e236e5c65465</a></p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1549252020-01-03T06:48:38ZKiefer Chang
<ul></ul><p>The mgr hangs when loading `diskprediction_local` module.</p>
<p>Some experiments:<br />- Move diskprediction_local module folder out of source tree to avoid loading it --> mgr starts.<br />- The new-created container has scipy 1.4.1. I temporarily replace it with scipy 1.3.2 --> mgr starts.</p>
<p>May be related to <a class="issue tracker-1 status-3 priority-6 priority-high2 closed" title="Bug: Test failure: test_diskprediction_local (tasks.mgr.test_module_selftest.TestModuleSelftest) (Resolved)" href="https://tracker.ceph.com/issues/42764">#42764</a>.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1549282020-01-03T08:18:51ZVolker Theile
<ul></ul><p>Kiefer, you made my day. I can confirm that removing the src/pybind/mgr/diskprediction_local/ directory fixes the issue. I assume that downgrading scipy will also fix it, too.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1557872020-01-15T12:59:20ZPatrick Seidensal
<ul></ul><p>I ran into this today and removing `diskprediction_local` from `src/pybind/mgr` resolved the issue for me.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1557892020-01-15T13:01:43ZPatrick Seidensal
<ul></ul><p>@Kiefer Chang</p>
<p>That explains why a new build of my container seems to have broken vstart! Thanks for the investigation.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1573622020-01-29T13:12:39ZLenz Grimmer
<ul></ul><p>FWIW, I ran into this issue myself a few days ago and was not able to determine the root cause. Downgrading scipy to version 1.3.2 fixed the issue for me.</p>
<p>We should investigate why newer versions of that module cause the diskprediction module to get stuck in initialization.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1573642020-01-29T13:17:00ZLenz Grimmer
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>mgr</i></li><li><strong>Subject</strong> changed from <i>vstart: ceph -h does not show manager module commands</i> to <i>mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versions</i></li><li><strong>Category</strong> changed from <i>ceph cli</i> to <i>diskprediction_cloud</i></li><li><strong>Regression</strong> changed from <i>No</i> to <i>Yes</i></li><li><strong>Severity</strong> changed from <i>3 - minor</i> to <i>2 - major</i></li></ul><p>Updated component and severity according to the latest findings.</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1608522020-03-12T08:10:00ZKiefer Chang
<ul></ul><p>gdb call trace of ceph-mgr when using with scipy 1.4.1.</p>
<pre>
(gdb) bt
#0 0x00007f1eefe96aaa in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1ef0582f6f in PyEval_RestoreThread () from /usr/lib64/libpython3.8.so.1.0
#2 0x00007f1ef0608c59 in PyGILState_Ensure () from /usr/lib64/libpython3.8.so.1.0
#3 0x00007f1ec3003e7d in ?? () from /usr/lib64/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so
#4 0x00007f1ec2f7dcb5 in ?? () from /usr/lib64/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so
#5 0x00007f1ec30027ee in ?? () from /usr/lib64/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so
#6 0x00007f1ec2ff204d in ?? () from /usr/lib64/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so
#7 0x00007f1ec2ff4f78 in PyInit_pypocketfft () from /usr/lib64/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so
#8 0x00007f1ef060e3cf in _PyImport_LoadDynamicModuleWithSpec () from /usr/lib64/libpython3.8.so.1.0
#9 0x00007f1ef060fc1d in ?? () from /usr/lib64/libpython3.8.so.1.0
#10 0x00007f1ef0516d97 in ?? () from /usr/lib64/libpython3.8.so.1.0
#11 0x00007f1ef055db52 in PyVectorcall_Call () from /usr/lib64/libpython3.8.so.1.0
</pre>
<p>NOTE: scipy.fft is a new submodule since scipy 1.4.0 (<a href="https://github.com/scipy/scipy/releases" class="external">release note</a>)</p> mgr - Bug #43447: mgr/diskprediction: diskprediction module fails to initialize with newer SciPy versionshttps://tracker.ceph.com/issues/43447?journal_id=1644662020-04-29T09:42:32ZKiefer Chang
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed" href="/issues/45147">Bug #45147</a>: Module 'diskprediction_local' takes forever to load</i> added</li></ul>