Project

General

Profile

Bug #43566

jenkins: ctest output size for failed test is too short

Added by Mykola Golub almost 2 years ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
% Done:

0%

Source:
Tags:
Backport:
Octopus, Pacific
Regression:
No
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We observe sporadic failures for rbd-mirror [1] and librbd [2] unit tests for jenkins make check. The problem is that the test's output buffer is too short and we don't see which test exactly fails to nail it down (trying to reproduce it locally rerunning the whole test set has not succeeded so far).

Can we increase somehow the output size? I suppose it can be tweaked by `--test-output-size-failed` ctest option [3]. Also, there is the cmake variable `CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE` [1], which according to the documentation is 300K.

Interestingly, the amd64 jenkins shows the whole fail test output, though I failed to find any related difference in ceph-build scripts between x86 and amd64. Probably, ctest is built with different default values there.

The easiest way to see the problem, I think, is to add something like `exit 1` at the end of `src/test/run-rbd-unit-tests.sh` to make it fail.

[1] https://tracker.ceph.com/issues/43274
[2] https://tracker.ceph.com/issues/43228
[3] https://cmake.org/cmake/help/v3.4/manual/ctest.1.html
[4] https://cmake.org/cmake/help/v3.4/variable/CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE.html


Related issues

Related to rbd - Bug #43274: unittest_rbd_mirror: Exception: SegFault Need More Info
Related to rbd - Bug #43228: run-rbd-unit-tests-0.sh fails Need More Info

History

#1 Updated by Nathan Cutler over 1 year ago

  • Subject changed from jenkins: ctest output size for failed test is too shot to jenkins: ctest output size for failed test is too short

#2 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #43274: unittest_rbd_mirror: Exception: SegFault added

#3 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #43228: run-rbd-unit-tests-0.sh fails added

#5 Updated by David Galloway 2 months ago

diff --git a/run-make-check.sh b/run-make-check.sh
index 7d64f968e12..98706b756eb 100755
--- a/run-make-check.sh
+++ b/run-make-check.sh
@@ -98,6 +98,9 @@ function main() {
     if [ $WITH_PMEM ]; then
         cmake_opts+=" -DWITH_RBD_RWL=ON -DWITH_SYSTEM_PMDK=ON" 
     fi
+    if in_jenkins; then
+        cmake_opts+=" -DCTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE=1024000" 
+    fi
     configure $cmake_opts $@
     build tests
     echo "make check: successful build on $(git rev-parse HEAD)" 

Resulted in:

CMake Warning:
  Manually-specified variables were not used by the project:

    CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE

#7 Updated by Kefu Chai 2 months ago

  • Status changed from New to Resolved

#8 Updated by David Galloway 2 months ago

  • Assignee set to Yehuda Sadeh
  • Backport set to Octopus, Pacific
  • Severity deleted (3 - minor)

#9 Updated by David Galloway 2 months ago

  • Assignee changed from Yehuda Sadeh to David Galloway
  • Pull request ID set to 42835

#10 Updated by David Galloway 2 months ago

  • Status changed from Resolved to Pending Backport

#11 Updated by David Galloway 2 months ago

I couldn't figure out the proper backport process. Perhaps because this isn't the Ceph tracker queue.

Anyway.

Octopus: https://github.com/ceph/ceph/pull/42849
Pacific: https://github.com/ceph/ceph/pull/42850

#12 Updated by Kefu Chai 2 months ago

  • Status changed from Pending Backport to Resolved

#13 Updated by Loïc Dachary 2 months ago

Perhaps because this isn't the Ceph tracker queue.

Right, backports are for projects under the Ceph project in redmine, reason why it was ignored. Maybe this specific issue should have been moved to the Ceph project since it ended up being fixed with a PR against Ceph.

#14 Updated by David Galloway 2 months ago

Loïc Dachary wrote:

Right, backports are for projects under the Ceph project in redmine, reason why it was ignored. Maybe this specific issue should have been moved to the Ceph project since it ended up being fixed with a PR against Ceph.

Fair point. I will keep that in mind for next time. Thanks, Loïc!

Also available in: Atom PDF