https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-07-13T15:31:10ZCeph rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1168762018-07-13T15:31:10ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/116876/diff?detail_id=115317">diff</a>)</li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1168772018-07-13T15:32:04ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/116877/diff?detail_id=115318">diff</a>)</li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1168892018-07-13T17:41:17ZMykola Golubmgolub@suse.com
<ul></ul><p>Kefu, why do you think that TestLibRBD.TestCopyPP fails to finish?</p>
<p>The log says that run-rbd-unit-tests.sh was running for 51.02 sec and failed. For me it rather looks like the test output was truncated and we can't see the real error.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1169342018-07-14T07:16:48ZKefu Chaitchaikov@gmail.com
<ul></ul><p>Mykola, i am not sure on this. it just appears to be so. and i think you are also right, the output could be truncated as well.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1169552018-07-16T06:44:09ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Subject</strong> changed from <i>TestLibRBD.TestCopyPP fails to finish in jenkin's "make check" run</i> to <i>run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" run</i></li></ul><p>another occurrence:<br /><pre>
98/157 Test #1: run-rbd-unit-tests.sh ...................***Failed 67.42 sec
++ dirname /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh
+ source /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/detect-build-env-vars.sh
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib ']'
++ echo 'Enivronment Variables Already Set'
Enivronment Variables Already Set
+ PATH=/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/src:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ unset RBD_FEATURES
+ unittest_librbd
seed 20923
[==========] Running 681 tests from 66 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from TestMockDeepCopyRequest
...
...
[----------] 13 tests from TestMockOperationSnapshotRemoveRequest
[ RUN ] TestMockOperationSnapshotRemoveRequest.Success
[ OK ] TestMockOperationSnapshotRemoveRequest.Success (5 ms)
...
...
[ RUN ] TestMockOperationSnapshotRemoveRequest.RemoveChildParentError
SKIPPING
[ OK ] TestMocErrors while running CTest
Build step 'Execute shell' marked build as failure
</pre></p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1169742018-07-16T12:29:30ZJason Dillamandillaman@redhat.com
<ul></ul><p>I've seen the log truncation issue lots of times. It seems to only affect the x86_64 builds, so I always try to look at the arm build logs for the full failure log.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1190112018-08-16T12:18:48ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p>marking it urgent as it happens very often now.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1190122018-08-16T13:16:58ZJason Dillamandillaman@redhat.com
<ul></ul><p>@Kefu: any clue / suggestion for fixing the test output under jenkins? The logs are clearly cut off so we cannot see the actual failure and this is a change to the jenkins environment within the past several months.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1190682018-08-17T13:33:51ZKefu Chaitchaikov@gmail.com
<ul></ul><p>Jason, what was the change?</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1190752018-08-17T16:18:58ZJason Dillamandillaman@redhat.com
<ul></ul><p>No clue -- that's the problem. The jenkins bots used to always include the full stdout but now they don't.</p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1190852018-08-18T16:40:18ZMykola Golubmgolub@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Backport</strong> set to <i>mimic,luminous</i></li></ul><p>Due to truncated jenkin's log I am not sure I reproduced the same case that is observed on jenkin, but I can make unittest_librbd sporadically fail locally when running it on one cpu. It reports a leaked mock ioctx for a "SKIPPED" mock test. A way to reproduce:<br /><pre>
maha:~/ceph/ceph.trociny/build% while RBD_FEATURES=0 taskset --cpu-list 1 unittest_librbd --gtest_filter=TestMockObjectMap.NonDetainedUpdate; do sleep 0; done
...
seed 18952
Note: Google Test filter = TestMockObjectMap.NonDetainedUpdate
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from TestMockObjectMap
2018-08-18 19:09:45.709 7fbe59ca3a00 -1 WARNING: all dangerous and experimental features are enabled.
[ RUN ] TestMockObjectMap.NonDetainedUpdate
Failed to load class: cephfs (lib/libcls_cephfs.so): lib/libcls_cephfs.so: undefined symbol: _ZTI10PGLSFilter
Failed to load class: hello (lib/libcls_hello.so): lib/libcls_hello.so: undefined symbol: _ZTI10PGLSFilter
Failed to load class: log (lib/libcls_log.so): lib/libcls_log.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer4listE
Failed to load class: lua (lib/libcls_lua.so): lib/libcls_lua.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer4listE
Failed to load class: refcount (lib/libcls_refcount.so): lib/libcls_refcount.so: undefined symbol: _Z14cls_cxx_removePv
Failed to load class: rgw (lib/libcls_rgw.so): lib/libcls_rgw.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer4listE
Failed to load class: sdk (lib/libcls_sdk.so): lib/libcls_sdk.so: undefined symbol: _Z14cls_cxx_removePv
Failed to load class: user (lib/libcls_user.so): lib/libcls_user.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer4listE
Failed to load class: version (lib/libcls_version.so): lib/libcls_version.so: undefined symbol: _Z19cls_gen_rand_base64Pci
SKIPPING
[ OK ] TestMockObjectMap.NonDetainedUpdate (8 ms)
[----------] 1 test from TestMockObjectMap (8 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (27 ms total)
[ PASSED ] 1 test.
/home/mgolub/ceph/ceph.trociny/src/test/librados_test_stub/MockTestMemRadosClient.h:66: ERROR: this mock object should be deleted but never is. Its address is @0x55c2350157e0.
/home/mgolub/ceph/ceph.trociny/src/test/librados_test_stub/MockTestMemIoCtxImpl.h:197: ERROR: this mock object (used in test TestMockObjectMap.NonDetainedUpdate) should be deleted but never is. Its address is @0x55c235203910.
ERROR: 2 leaked mock objects found at program exit.
</pre></p>
<p>I traced this to a race when completing `image::CreateRequest`: m_on_finish callback is called (and test finishes) before the request (and data ioctx) is destroyed.</p>
<p>PR: <a class="external" href="https://github.com/ceph/ceph/pull/23639">https://github.com/ceph/ceph/pull/23639</a></p> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1193002018-08-22T21:09:39ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Assignee</strong> set to <i>Mykola Golub</i></li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1193012018-08-22T21:10:22ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1193062018-08-22T21:11:03ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/27060">Backport #27060</a>: mimic: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" run</i> added</li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1193082018-08-22T21:11:06ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/27061">Backport #27061</a>: luminous: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" run</i> added</li></ul> rbd - Bug #24910: run-rbd-unit-tests.sh test fails to finish in jenkin's "make check" runhttps://tracker.ceph.com/issues/24910?journal_id=1212782018-09-24T11:16:25ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>