Project

General

Profile

Bug #46669

make check: unittest_rbd_mirror (Child aborted): failed, despite all tests passed

Added by Sebastian Wagner 3 months ago. Updated 29 days ago.

Status:
Resolved
Priority:
High
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus,octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

https://jenkins.ceph.com/job/ceph-pull-requests/56114/consoleFull#1884892015e840cee4-f4a4-4183-81dd-42855615f2c1

I really can't find any failed tests in this run, the only thing coming close to a failure is:

[----------] 11 tests from TestMockImageReplayer
[ RUN      ] TestMockImageReplayer.StartStop
Failed to load class: cas (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cas.so: undefined symbol: _Z13cls_has_chunkPvNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Failed to load class: cmpomap (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_cmpomap.so: undefined symbol: _Z28cls_cxx_map_get_vals_by_keysPvRKSt3setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4lessIS6_ESaIS6_EEPSt3mapIS6_N4ceph6buffer7v15_2_04listES8_SaISt4pairIKS6_SH_EEE
Failed to load class: log (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_log.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE
Failed to load class: rgw (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_rgw.so: undefined symbol: _Z19cls_current_versionPv
Failed to load class: user (/home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so): /home/jenkins-build/build/workspace/ceph-pull-requests/build/lib/libcls_user.so: undefined symbol: _Z24cls_cxx_map_write_headerPvPN4ceph6buffer7v15_2_04listE
[       OK ] TestMockImageReplayer.StartStop (181 ms)

The final messages are:

[----------] 5 tests from TestImageSync
[ RUN      ] TestImageSync.Empty
[       OK ] TestImageSync.Empty (219 ms)
[ RUN      ] TestImageSync.Simple
[       OK ] TestImageSync.Simple (205 ms)
[ RUN      ] TestImageSync.Resize
[       OK ] TestImageSync.Resize (211 ms)
[ RUN      ] TestImageSync.Discard
[       OK ] TestImageSync.Discard (219 ms)
[ RUN      ] TestImageSync.SnapshotStress
[       OK ] TestImageSync.SnapshotStress (434 ms)
[----------] 5 tests from TestImageSync (1288 ms total)

[----------] 2 tests from TestInstanceWatcher
terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided

195/204 Test #114: readable.sh ...............................   Passed  190.71 sec
196/204 Test   #9: run-rbd-unit-tests-61.sh ..................   Passed  198.20 sec
197/204 Test #113: check-generated.sh ........................   Passed  214.48 sec
198/204 Test   #2: run-tox-mgr-dashboard .....................   Passed  227.39 sec
199/204 Test #200: unittest_btree_lba_manager ................   Passed  211.92 sec
200/204 Test  #10: run-rbd-unit-tests-109.sh .................   Passed  254.11 sec
201/204 Test  #11: run-rbd-unit-tests-127.sh .................   Passed  255.33 sec
202/204 Test  #29: unittest_bufferlist .......................   Passed  295.69 sec
203/204 Test #142: mgr-dashboard-frontend-unittests ..........   Passed  291.19 sec
204/204 Test #126: unittest_erasure_code_shec_all ............   Passed  503.75 sec

99% tests passed, 1 tests failed out of 204

Total Test time (real) = 510.19 sec

The following tests FAILED:
    190 - unittest_rbd_mirror (Child aborted)

Related issues

Copied to rbd - Backport #46982: octopus: make check: unittest_rbd_mirror (Child aborted): failed, despite all tests passed Resolved
Copied to rbd - Backport #46983: nautilus: make check: unittest_rbd_mirror (Child aborted): failed, despite all tests passed Resolved

History

#2 Updated by Sebastian Wagner 3 months ago

  • Priority changed from Normal to High
  • Source set to Q/A

#3 Updated by Sebastian Wagner 3 months ago

Looks like we now have totally random failures:

https://jenkins.ceph.com/job/ceph-pull-requests/56284/consoleFull#964930372e840cee4-f4a4-4183-81dd-42855615f2c1

[ RUN      ] TestMockPoolWatcher.RegisterWatcherMissing
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/rbd_mirror/test_mock_PoolWatcher.cc:475: Failure
Expected equality of these values:
  -2
  ctx.wait()
    Which is: 0
[  FAILED  ] TestMockPoolWatcher.RegisterWatcherMissing (32 ms)
...
[  FAILED  ] TestMockPoolWatcher.RegisterWatcherMissing

 1 FAILED TEST

#4 Updated by Jason Dillaman 3 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Jason Dillaman
  • Backport set to nautilus,octopus
  • Pull request ID set to 36479

I cannot repeat the "Resource deadlock avoided" -- hopefully that was addressed by [1]

[1] https://github.com/ceph/ceph/pull/36287

#5 Updated by Mykola Golub 3 months ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Nathan Cutler 2 months ago

  • Copied to Backport #46982: octopus: make check: unittest_rbd_mirror (Child aborted): failed, despite all tests passed added

#7 Updated by Nathan Cutler 2 months ago

  • Copied to Backport #46983: nautilus: make check: unittest_rbd_mirror (Child aborted): failed, despite all tests passed added

#8 Updated by Mykola Golub 2 months ago

Jason Dillaman wrote:

I cannot repeat the "Resource deadlock avoided" -- hopefully that was addressed by [1]

[1] https://github.com/ceph/ceph/pull/36287

It looks like it was not. It is still observable on a fresh master [1].

[1] https://jenkins.ceph.com/job/ceph-pull-requests/57817/console

#10 Updated by Nathan Cutler 29 days ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF