Bug #43566
closedjenkins: ctest output size for failed test is too short
0%
Description
We observe sporadic failures for rbd-mirror [1] and librbd [2] unit tests for jenkins make check. The problem is that the test's output buffer is too short and we don't see which test exactly fails to nail it down (trying to reproduce it locally rerunning the whole test set has not succeeded so far).
Can we increase somehow the output size? I suppose it can be tweaked by `--test-output-size-failed` ctest option [3]. Also, there is the cmake variable `CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE` [1], which according to the documentation is 300K.
Interestingly, the amd64 jenkins shows the whole fail test output, though I failed to find any related difference in ceph-build scripts between x86 and amd64. Probably, ctest is built with different default values there.
The easiest way to see the problem, I think, is to add something like `exit 1` at the end of `src/test/run-rbd-unit-tests.sh` to make it fail.
[1] https://tracker.ceph.com/issues/43274
[2] https://tracker.ceph.com/issues/43228
[3] https://cmake.org/cmake/help/v3.4/manual/ctest.1.html
[4] https://cmake.org/cmake/help/v3.4/variable/CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE.html
Updated by Nathan Cutler about 4 years ago
- Subject changed from jenkins: ctest output size for failed test is too shot to jenkins: ctest output size for failed test is too short
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #43274: unittest_rbd_mirror: Exception: SegFault added
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #43228: run-rbd-unit-tests-0.sh fails added
Updated by David Galloway over 2 years ago
I guess that would get defined here? https://github.com/ceph/ceph/blob/master/run-make-check.sh#L91
Updated by David Galloway over 2 years ago
diff --git a/run-make-check.sh b/run-make-check.sh
index 7d64f968e12..98706b756eb 100755
--- a/run-make-check.sh
+++ b/run-make-check.sh
@@ -98,6 +98,9 @@ function main() {
if [ $WITH_PMEM ]; then
cmake_opts+=" -DWITH_RBD_RWL=ON -DWITH_SYSTEM_PMDK=ON"
fi
+ if in_jenkins; then
+ cmake_opts+=" -DCTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE=1024000"
+ fi
configure $cmake_opts $@
build tests
echo "make check: successful build on $(git rev-parse HEAD)"
Resulted in:
CMake Warning:
Manually-specified variables were not used by the project:
CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE
Updated by David Galloway over 2 years ago
Updated by David Galloway over 2 years ago
- Assignee set to Yehuda Sadeh
- Backport set to Octopus, Pacific
- Severity deleted (
3 - minor)
Updated by David Galloway over 2 years ago
- Assignee changed from Yehuda Sadeh to David Galloway
- Pull request ID set to 42835
Updated by David Galloway over 2 years ago
- Status changed from Resolved to Pending Backport
Updated by David Galloway over 2 years ago
I couldn't figure out the proper backport process. Perhaps because this isn't the Ceph tracker queue.
Anyway.
Octopus: https://github.com/ceph/ceph/pull/42849
Pacific: https://github.com/ceph/ceph/pull/42850
Updated by Kefu Chai over 2 years ago
- Status changed from Pending Backport to Resolved
Updated by Loïc Dachary over 2 years ago
Perhaps because this isn't the Ceph tracker queue.
Right, backports are for projects under the Ceph project in redmine, reason why it was ignored. Maybe this specific issue should have been moved to the Ceph project since it ended up being fixed with a PR against Ceph.
Updated by David Galloway over 2 years ago
Loïc Dachary wrote:
Right, backports are for projects under the Ceph project in redmine, reason why it was ignored. Maybe this specific issue should have been moved to the Ceph project since it ended up being fixed with a PR against Ceph.
Fair point. I will keep that in mind for next time. Thanks, Loïc!