Project

General

Profile

Actions
Copy link

Bug #17446

closed

assert failure in run-rbd-unit-tests.sh

Added by Kefu Chai over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

see https://jenkins.ceph.com/job/ceph-pull-requests/12079/consoleFull#1073173703d63714d2-c8d8-41fc-a9d4-8dee30be4c32

8: [ RUN      ] TestMirroring.EnableImageMirror_In_MirrorModeImage
8: /var/ceph/ceph/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f8ae2fe0880 time 2016-09-30 16:06:28.304875
8: /var/ceph/ceph/src/common/AsyncOpTracker.cc: 14: FAILED assert(m_pending_ops == 0)
8:  ceph version Development (no_version)
8:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x556b102b7bda]
8:  2: (AsyncOpTracker::~AsyncOpTracker()+0x58) [0x556b10283288]
8:  3: (journal::JournalMetadata::~JournalMetadata()+0x96) [0x556b0feb9052]
8:  4: (journal::JournalMetadata::~JournalMetadata()+0x18) [0x556b0feb9170]
8:  5: (RefCountedObject::put()+0x197) [0x556b0feabc59]
8:  6: (journal::Journaler::~Journaler()+0x69) [0x556b0feafd11]
8:  7: (librbd::Journal<librbd::ImageCtx>::get_tag_owner(librados::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_s
tring<char, std::char_traits<char>, std::allocator<char> >*)+0x29c) [0x556b0fc3e4ab]                                                                                              8:  8: (librbd::Journal<librbd::ImageCtx>::is_tag_owner(librados::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool*)+0x47) [0x556b0f
c3df43]                                                                                                                                                                           8:  9: (librbd::Journal<librbd::ImageCtx>::is_tag_owner(librbd::ImageCtx*, bool*)+0x35) [0x556b0fc3defa]
8:  10: (librbd::mirror_image_get_info(librbd::ImageCtx*, librbd::mirror_image_info_t*, unsigned long)+0x3a8) [0x556b10120b4b]
8:  11: (librbd::Image::mirror_image_get_info(librbd::mirror_image_info_t*, unsigned long)+0x36) [0x556b10091ba2]
8:  12: (TestMirroring::check_mirror_image_enable(rbd_mirror_mode_t, unsigned long, int, rbd_mirror_image_state_t)+0x661) [0x556b100437d1]
8:  13: (TestMirroring_EnableImageMirror_In_MirrorModeImage_Test::TestBody()+0x43) [0x556b1003c53f]
8:  14: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x65) [0x556b1025d1b6]
8:  15: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x5a) [0x556b10257e5d]
8:  16: (testing::Test::Run()+0xd4) [0x556b1023cc36]
8:  17: (testing::TestInfo::Run()+0x10a) [0x556b1023d55c]
8:  18: (testing::TestCase::Run()+0x107) [0x556b1023dbe5]
8:  19: (testing::internal::UnitTestImpl::RunAllTests()+0x2d2) [0x556b10244d94]
8:  20: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTest
Impl::*)(), char const*)+0x65) [0x556b1025e0eb]
8:  21: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImp
l::*)(), char const*)+0x5a) [0x556b10258beb]
8:  22: (testing::UnitTest::Run()+0xc6) [0x556b10243958]
8:  23: (RUN_ALL_TESTS()+0x11) [0x556b0faf9204]
8:  24: (main()+0x130) [0x556b0faf9096]
8:  25: (__libc_start_main()+0xf1) [0x7f8ad68ba2b1]
8:  26: (_start()+0x2a) [0x556b0faf422a]
8:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
8: /var/ceph/ceph/src/test/run-rbd-unit-tests.sh: line 11: 38569 Aborted                 RBD_FEATURES=$i unittest_librbd
1/1 Test #8: run-rbd-unit-tests.sh ............***Failed   65.52 sec

Related issues 1 (0 open — 1 closed)

Copied to rbd - Backport #17506: jewel: assert failure in run-rbd-unit-tests.shResolvedJason DillamanActions
Actions
Copy link
 #1

Updated by Kefu Chai over 7 years ago

  • Project changed from Ceph to rbd

not consistently reproducible.

Actions
Copy link
 #2

Updated by Alexey Sheplyakov over 7 years ago

rbd unit tests fail for me too (more often than not) with a somewhat similar backtrace:

[----------] 12 tests from TestJournaler
[ RUN ] TestJournaler.Create
[ OK ] TestJournaler.Create (0 ms)
[ RUN ] TestJournaler.CreateDuplicate
2016-09-30 12:58:36.049731 7f31d17fa700 0 <cls> /home/asheplyakov/work/ceph/ceph/src/cls/journal/cls_journal.cc:333: journal already exists
[ OK ] TestJournaler.CreateDuplicate (0 ms)
[ RUN ] TestJournaler.CreateInvalidParams
[ OK ] TestJournaler.CreateInvalidParams (0 ms)
[ RUN ] TestJournaler.Init
/home/asheplyakov/work/ceph/ceph/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f31e4903e00 time 2016-09-30 12:58:36.049900
/home/asheplyakov/work/ceph/ceph/src/common/AsyncOpTracker.cc: 14: FAILED assert(m_pending_ops 0)
ceph version v11.0.0-2803-g9ff377f (9ff377f674d20e7d3f0e40dcb830b68bfa05c79a)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x55c7dec93bbb]
2: (()+0x315c32) [0x55c7dec86c32]
3: (journal::JournalMetadata::~JournalMetadata()+0x56) [0x55c7dec24926]
4: (journal::JournalMetadata::~JournalMetadata()+0x9) [0x55c7dec24af9]
5: (RefCountedObject::put()+0xe7) [0x55c7dec16e37]
6: (journal::Journaler::~Journaler()+0x24) [0x55c7dec18774]
7: (TestJournaler::register_client(std::string const&, std::string const&)+0x314) [0x55c7deb4bc64]
8: (TestJournaler_Init_Test::TestBody()+0x190) [0x55c7deb43db0]
9: (void testing::internal::HandleExceptionsInMethodIfSupported&lt;testing::Test, void&gt;(testing::Test*, void (testing::Test::*)(), char const*)+0x33) [0x55c7dec12c93]
10: (testing::Test::Run()+0xb7) [0x55c7dec05e87]
11: (testing::TestInfo::Run()+0x9e) [0x55c7dec05f2e]
12: (testing::TestCase::Run()+0xa5) [0x55c7dec06035]
13: (testing::internal::UnitTestImpl::RunAllTests()+0x248) [0x55c7dec062e8]
14: (testing::UnitTest::Run()+0x54) [0x55c7dec065a4]
15: (main()+0xa9) [0x55c7deaddb19]
16: (__libc_start_main()+0xf5) [0x7f31d9e5ef45]
17: (()+0x1bf447) [0x55c7deb30447]
NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.
2016-09-30 12:58:36.051079 7f31e4903e00 -1 /home/asheplyakov/work/ceph/ceph/src/common/AsyncOpTracker.cc: In function 'AsyncOpTracker::~AsyncOpTracker()' thread 7f31e4903e00
/home/asheplyakov/work/ceph/ceph/src/common/AsyncOpTracker.cc: 14: FAILED assert(m_pending_ops 0)

Actions
Copy link
 #3

Updated by Alexey Sheplyakov over 7 years ago

The following patch seems to fix (or at least work around) the issue for me:

diff --git a/src/journal/Journaler.cc b/src/journal/Journaler.cc
index 108e428..f6a37f3 100644

--- a/src/journal/Journaler.cc
+++ b/src/journal/Journaler.cc
@@ -105,7 +105,7 @@ void Journaler::set_up(ContextWQ *work_queue, SafeTimer *timer,
 Journaler::~Journaler() {
   if (m_metadata != nullptr) {
     assert(!m_metadata->is_initialized());
-    m_metadata->put();
+    shut_down();
     m_metadata = nullptr;
   }
   assert(m_trimmer == nullptr);

I'm not sure if it's correct, though

Actions
Copy link
 #4

Updated by Jason Dillaman over 7 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #5

Updated by Jason Dillaman over 7 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to jewel
Actions
Copy link
 #6

Updated by Nathan Cutler over 7 years ago

  • Copied to Backport #17506: jewel: assert failure in run-rbd-unit-tests.sh added
Actions
Copy link
 #7

Updated by Loïc Dachary over 7 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF