Ceph » rbd

Possible race due to the test does not properly wait for release leader lock to complete before initiating shutdown:

 RUN      ] TestMockLeaderWatcher.InitReleaseShutdown
/home/jenkins-build/build/workspace/ceph-dashboard-pull-requests/src/common/Mutex.cc: In function 'void Mutex::lock(bool)' thread 7f529d29a700 time 2019-01-11 22:43:22.859667
/home/jenkins-build/build/workspace/ceph-dashboard-pull-requests/src/common/Mutex.cc: 79: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) nautilus (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f52af1d4d4a]
 2: (()+0x1175fcc) [0x7f52af1d4fcc]
 3: (Mutex::lock(bool)+0x76) [0x7f52af1616e6]
 4: (std::lock_guard<Mutex>::lock_guard(Mutex&)+0x2f) [0x1471ea3]
 5: /home/jenkins-build/build/workspace/ceph-dashboard-pull-requests/build/bin/unittest_rbd_mirror() [0x16c5c14]
 6: /home/jenkins-build/build/workspace/ceph-dashboard-pull-requests/build/bin/unittest_rbd_mirror() [0x16c968a]
 7: (Context::complete(int)+0x27) [0x1468bb9]
 8: (ContextWQ::process(Context*)+0xdd) [0x199f899]
 9: (ThreadPool::PointerWQ<Context>::_void_process(void*, ThreadPool::TPHandle&)+0x32) [0x19a165a]
 10: (ThreadPool::worker(ThreadPool::WorkThread*)+0x739) [0x7f52af199cdb]
 11: (ThreadPool::WorkThread::entry()+0x23) [0x7f52af19de09]
 12: (Thread::entry_wrapper()+0x78) [0x7f52af1753a2]
 13: (Thread::_entry_func(void*)+0x18) [0x7f52af175320]
 14: (()+0x7dd5) [0x7f52b95c4dd5]
 15: (clone()+0x6d) [0x7f52ab440ead]

#0  0x00007fef5991508b in raise () from /lib64/libc.so.6
#1  0x00007fef598fe4e9 in abort () from /lib64/libc.so.6
#2  0x00007fef5bd6826d in ceph::__ceph_assert_fail (assertion=0x7fef5c5301ab "r == 0", file=0x7fef5c530180 "/home/mgolub/ceph/ceph/src/common/Mutex.cc", line=79,
    func=0x7fef5c530210 <Mutex::lock(bool)::__PRETTY_FUNCTION__> "void Mutex::lock(bool)") at /home/mgolub/ceph/ceph/src/common/assert.cc:73
#3  0x00007fef5bd68320 in ceph::__ceph_assert_fail (ctx=...) at /home/mgolub/ceph/ceph/src/common/assert.cc:78
#4  0x00007fef5bcf528a in Mutex::lock (this=0x7ffe94d94290, no_lockdep=false) at /home/mgolub/ceph/ceph/src/common/Mutex.cc:79
#5  0x000055d9b80a647d in std::lock_guard<Mutex>::lock_guard (this=0x7fee727f7728, __m=...) at /usr/include/c++/8/bits/std_mutex.h:162
#6  0x000055d9b82e939f in rbd::mirror::LeaderWatcher<librbd::(anonymous namespace)::MockTestImageCtx>::handle_release_leader_lock (this=0x7ffe94d94070, r=0)
    at /home/mgolub/ceph/ceph/src/tools/rbd_mirror/LeaderWatcher.cc:684
#7  0x000055d9b82f0ff0 in librbd::util::detail::C_CallbackAdapter<rbd::mirror::LeaderWatcher<librbd::(anonymous namespace)::MockTestImageCtx>, &rbd::mirror::LeaderWatcher<librbd::(anonymous namespace)::MockTestImageCtx>::handle_release_leader_lock>::finish (this=0x55d9b998a340, r=0) at /home/mgolub/ceph/ceph/src/librbd/Utils.h:56
#8  0x000055d9b809cf05 in Context::complete (this=0x55d9b998a340, r=0) at /home/mgolub/ceph/ceph/src/include/Context.h:77
#9  0x000055d9b85cd9bf in ContextWQ::process (this=0x55d9b99a2de0, ctx=0x55d9b998a340) at /home/mgolub/ceph/ceph/src/common/WorkQueue.h:620
#10 0x000055d9b85cf8b8 in ThreadPool::PointerWQ<Context>::_void_process (this=0x55d9b99a2de0, item=0x55d9b998a340, handle=...) at /home/mgolub/ceph/ceph/src/common/WorkQueue.h:410
#11 0x00007fef5bd2ce6b in ThreadPool::worker (this=0x55d9b9937580, wt=0x55d9b9985a70) at /home/mgolub/ceph/ceph/src/common/WorkQueue.cc:118
#12 0x00007fef5bd30fb3 in ThreadPool::WorkThread::entry (this=0x55d9b9985a70) at /home/mgolub/ceph/ceph/src/common/WorkQueue.h:459
#13 0x00007fef5bd08804 in Thread::entry_wrapper (this=0x55d9b9985a70) at /home/mgolub/ceph/ceph/src/common/Thread.cc:84
#14 0x00007fef5bd08782 in Thread::_entry_func (arg=0x55d9b9985a70) at /home/mgolub/ceph/ceph/src/common/Thread.cc:71
#15 0x00007fef6549b554 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fef599d7ccf in clone () from /lib64/libc.so.6