Bug #54292
run-rbd-unit-tests-127.sh times out on Jenkins "make check" runs
0%
Description
This test fails sometimes on PRs, and then passes upon re-runing make check. It's possible that the timeout threshold needs to be increased.
See attached text file for the console log of this build if the link below expires.
https://jenkins.ceph.com/job/ceph-pull-requests/90258/
275/275 Test #34: run-rbd-unit-tests-127.sh .................***Timeout 3600.11 sec
++ dirname /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh
+ source /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/detect-build-env-vars.sh
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib ']'
++ echo 'Enivronment Variables Already Set'
Enivronment Variables Already Set
+ PATH=/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/src:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ '[' 1 = 0 ']'
+ '[' 1 = 1 -a 127 = N ']'
+ TESTS=127
+ for i in ${TESTS}
+ RBD_FEATURES=127
+ unittest_librbd
seed 3874247
[==========] Running 1148 tests from 105 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from TestIOBlockGuard
[ RUN ] TestIOBlockGuard.NonDetainedOps
Failed to load class: cas (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so: undefined symbol: _Z26cls_get_manifest_ref_countPvNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Failed to load class: cmpomap (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so: undefined symbol: _Z28cls_cxx_map_get_vals_by_keysPvRKSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS6_ESaIS6_EEPSt3mapIS6_N4ceph6buffer7v15_2_04listES8_SaISt4pairIKS6_SH_EEE
Failed to load class: fifo (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_fifo.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_fifo.so: undefined symbol: _Z20cls_gen_random_bytesPci
Failed to load class: log (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE
Failed to load class: rgw (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so: undefined symbol: _Z19cls_current_versionPv
Failed to load class: user (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE
History
#1 Updated by Ilya Dryomov about 2 years ago
Looks like TestLibRBD.QuiesceWatchError is the culprit:
[ RUN ] TestLibRBD.QuiesceWatch using new format! creating image with stripe unit: 65536, stripe count: 16 [ OK ] TestLibRBD.QuiesceWatch (363 ms) [ RUN ] TestLibRBD.QuiesceWatchPP using new format! [ OK ] TestLibRBD.QuiesceWatchPP (308 ms) [ RUN ] TestLibRBD.QuiesceWatchError using new format! 99% tests passed, 1 tests failed out of 275 Total Test time (real) = 3604.58 sec The following tests FAILED: 34 - run-rbd-unit-tests-127.sh (Timeout)
#2 Updated by Ilya Dryomov about 2 years ago
It's possible that the timeout threshold needs to be increased.
It definitely shouldn't take more than an hour. These tests usually finish within 5-7 minutes.
#3 Updated by Laura Flores about 2 years ago
- Related to Bug #54302: unittest_rbd_mirror fails due to SEGFAULT exception on Jenkins "make check" runs added
#4 Updated by Laura Flores about 2 years ago
Thanks for the insight, Ilya. FYI this bug may be happening less frequently than I thought. There is another rbd unittest that also sometimes fails in make check (see related issue), but from what I have observed, they are not happening terribly often. Although the frequency is low, I sent the priority to "high" because these failures sometimes block PR merges.
#5 Updated by Ilya Dryomov about 2 years ago
- Related to deleted (Bug #54302: unittest_rbd_mirror fails due to SEGFAULT exception on Jenkins "make check" runs)