Project

General

Profile

Bug #54292

run-rbd-unit-tests-127.sh times out on Jenkins "make check" runs

Added by Laura Flores about 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This test fails sometimes on PRs, and then passes upon re-runing make check. It's possible that the timeout threshold needs to be increased.
See attached text file for the console log of this build if the link below expires.

https://jenkins.ceph.com/job/ceph-pull-requests/90258/

275/275 Test  #34: run-rbd-unit-tests-127.sh .................***Timeout 3600.11 sec
++ dirname /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/run-rbd-unit-tests.sh
+ source /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/detect-build-env-vars.sh
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin ']'
++ '[' -n /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib ']'
++ echo 'Enivronment Variables Already Set'
Enivronment Variables Already Set
+ PATH=/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin:/home/jenkins-build/build/workspace/ceph-pull-requests/src:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ '[' 1 = 0 ']'
+ '[' 1 = 1 -a 127 = N ']'
+ TESTS=127
+ for i in ${TESTS}
+ RBD_FEATURES=127
+ unittest_librbd
seed 3874247
[==========] Running 1148 tests from 105 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from TestIOBlockGuard
[ RUN      ] TestIOBlockGuard.NonDetainedOps
Failed to load class: cas (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so: undefined symbol: _Z26cls_get_manifest_ref_countPvNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Failed to load class: cmpomap (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so: undefined symbol: _Z28cls_cxx_map_get_vals_by_keysPvRKSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS6_ESaIS6_EEPSt3mapIS6_N4ceph6buffer7v15_2_04listES8_SaISt4pairIKS6_SH_EEE
Failed to load class: fifo (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_fifo.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_fifo.so: undefined symbol: _Z20cls_gen_random_bytesPci
Failed to load class: log (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE
Failed to load class: rgw (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so: undefined symbol: _Z19cls_current_versionPv
Failed to load class: user (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE

run-rbd-unit-tests-127-timeout.txt View (628 KB) Laura Flores, 02/15/2022 10:47 PM

History

#1 Updated by Ilya Dryomov about 2 years ago

Looks like TestLibRBD.QuiesceWatchError is the culprit:

[ RUN      ] TestLibRBD.QuiesceWatch
using new format!
creating image with stripe unit: 65536, stripe count: 16
[       OK ] TestLibRBD.QuiesceWatch (363 ms)
[ RUN      ] TestLibRBD.QuiesceWatchPP
using new format!
[       OK ] TestLibRBD.QuiesceWatchPP (308 ms)
[ RUN      ] TestLibRBD.QuiesceWatchError
using new format!

99% tests passed, 1 tests failed out of 275

Total Test time (real) = 3604.58 sec

The following tests FAILED:
     34 - run-rbd-unit-tests-127.sh (Timeout)

#2 Updated by Ilya Dryomov about 2 years ago

It's possible that the timeout threshold needs to be increased.

It definitely shouldn't take more than an hour. These tests usually finish within 5-7 minutes.

#3 Updated by Laura Flores about 2 years ago

  • Related to Bug #54302: unittest_rbd_mirror fails due to SEGFAULT exception on Jenkins "make check" runs added

#4 Updated by Laura Flores about 2 years ago

Thanks for the insight, Ilya. FYI this bug may be happening less frequently than I thought. There is another rbd unittest that also sometimes fails in make check (see related issue), but from what I have observed, they are not happening terribly often. Although the frequency is low, I sent the priority to "high" because these failures sometimes block PR merges.

#5 Updated by Ilya Dryomov about 2 years ago

  • Related to deleted (Bug #54302: unittest_rbd_mirror fails due to SEGFAULT exception on Jenkins "make check" runs)

Also available in: Atom PDF