Project

General

Profile

Actions
Copy link

Bug #24166

closed

mgr: deadlock between PyModuleRegistry and ms_dispatch and ...

Added by Sage Weil almost 6 years ago. Updated over 5 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84558b8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381675 in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  MgrStandby::ms_dispatch (this=0x7ffdf8453770, m=0x55712f620a00) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:411
#5  0x00007f824f44de02 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0
#6  0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#7  0x00007f824e6dc6ba in start_thread (arg=0x7f8245230700) at pthread_create.c:333
#8  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84559f8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381b5d in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  PyModuleRegistry::get_modules[abi:cxx11]() const (this=0x7ffdf84559d0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.h:70
#5  MgrStandby::send_beacon (this=this@entry=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:185
#6  0x000055712c382b3f in MgrStandby::tick (this=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:239
#7  0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#8  FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#9  0x000055712c3332d9 in Context::complete (this=0x55712eb84570, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#10 0x00007f824f3d1aab in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#11 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#12 0x00007f824e6dc6ba in start_thread (arg=0x7f8241a29700) at pthread_create.c:333
#13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 33 (Thread 0x7f8235e08700 (LWP 2489629)):
#0  0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x000055712c371258 in PyModuleRunner::shutdown (this=this@entry=0x55712d3a2f40) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:82
#14 0x000055712c3616ac in StandbyPyModules::shutdown (this=0x55712d3dd8c0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/StandbyPyModules.cc:56
#15 0x000055712c36c623 in PyModuleRegistry::active_start (this=0x7ffdf84559d0, ds=..., cs=..., kv_store=std::map with 5 elements = {...}, mc=..., clog_=std::shared_ptr (count 9, weak 0) 0x55712d41b7b0, objecter_=..., client_=..., 
    f=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.cc:187
#16 0x000055712c3946b7 in Mgr::init (this=0x55712ed52000) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:263
#17 0x000055712c394a1c in Mgr::<lambda(int)>::operator() (r=<optimized out>, __closure=0x55712eb856f0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:148
#18 boost::detail::function::void_function_obj_invoker1<Mgr::background_init(Context*)::<lambda(int)>, void, int>::invoke(boost::detail::function::function_buffer &, int) (function_obj_ptr=..., a0=<optimized out>)
    at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:159
#19 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#20 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#21 0x000055712c3332d9 in Context::complete (this=0x55712eb856e0, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#22 0x00007f824f3d36ce in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#23 0x00007f824e6dc6ba in start_thread (arg=0x7f8235e08700) at pthread_create.c:333
#24 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

on lab cluster

Actions
Copy link

Also available in: Atom PDF