Bug #24166
mgr: deadlock between PyModuleRegistry and ms_dispatch and ...
Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
(gdb) bt #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84558b8) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0 #3 0x000055712c381675 in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115 #4 MgrStandby::ms_dispatch (this=0x7ffdf8453770, m=0x55712f620a00) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:411 #5 0x00007f824f44de02 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0 #6 0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #7 0x00007f824e6dc6ba in start_thread (arg=0x7f8245230700) at pthread_create.c:333 #8 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 (gdb) bt #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84559f8) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0 #3 0x000055712c381b5d in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115 #4 PyModuleRegistry::get_modules[abi:cxx11]() const (this=0x7ffdf84559d0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.h:70 #5 MgrStandby::send_beacon (this=this@entry=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:185 #6 0x000055712c382b3f in MgrStandby::tick (this=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:239 #7 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768 #8 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522 #9 0x000055712c3332d9 in Context::complete (this=0x55712eb84570, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77 #10 0x00007f824f3d1aab in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0 #11 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #12 0x00007f824e6dc6ba in start_thread (arg=0x7f8241a29700) at pthread_create.c:333 #13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 33 (Thread 0x7f8235e08700 (LWP 2489629)): #0 0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #2 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #3 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #4 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x000055712c371258 in PyModuleRunner::shutdown (this=this@entry=0x55712d3a2f40) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:82 #14 0x000055712c3616ac in StandbyPyModules::shutdown (this=0x55712d3dd8c0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/StandbyPyModules.cc:56 #15 0x000055712c36c623 in PyModuleRegistry::active_start (this=0x7ffdf84559d0, ds=..., cs=..., kv_store=std::map with 5 elements = {...}, mc=..., clog_=std::shared_ptr (count 9, weak 0) 0x55712d41b7b0, objecter_=..., client_=..., f=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.cc:187 #16 0x000055712c3946b7 in Mgr::init (this=0x55712ed52000) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:263 #17 0x000055712c394a1c in Mgr::<lambda(int)>::operator() (r=<optimized out>, __closure=0x55712eb856f0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:148 #18 boost::detail::function::void_function_obj_invoker1<Mgr::background_init(Context*)::<lambda(int)>, void, int>::invoke(boost::detail::function::function_buffer &, int) (function_obj_ptr=..., a0=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:159 #19 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768 #20 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522 #21 0x000055712c3332d9 in Context::complete (this=0x55712eb856e0, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77 #22 0x00007f824f3d36ce in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0 #23 0x00007f824e6dc6ba in start_thread (arg=0x7f8235e08700) at pthread_create.c:333 #24 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
on lab cluster
History
#1 Updated by Sage Weil almost 6 years ago
all threads
(gdb) thr app all bt Thread 35 (Thread 0x7f8234e06700 (LWP 2489631)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f44fa10 in DispatchQueue::run_local_delivery() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f4ec10d in DispatchQueue::LocalDeliveryThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f8234e06700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 34 (Thread 0x7f8235607700 (LWP 2489630)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f44d907 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f8235607700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 33 (Thread 0x7f8235e08700 (LWP 2489629)): #0 0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #2 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #3 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #4 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x000055712c371258 in PyModuleRunner::shutdown (this=this@entry=0x55712d3a2f40) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:82 #14 0x000055712c3616ac in StandbyPyModules::shutdown (this=0x55712d3dd8c0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/StandbyPyModules.cc:56 #15 0x000055712c36c623 in PyModuleRegistry::active_start (this=0x7ffdf84559d0, ds=..., cs=..., kv_store=std::map with 5 elements = {...}, mc=..., clog_=std::shared_ptr (count 9, weak 0) 0x55712d41b7b0, objecter_=..., client_=..., f=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.cc:187 #16 0x000055712c3946b7 in Mgr::init (this=0x55712ed52000) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:263 #17 0x000055712c394a1c in Mgr::<lambda(int)>::operator() (r=<optimized out>, __closure=0x55712eb856f0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:148 #18 boost::detail::function::void_function_obj_invoker1<Mgr::background_init(Context*)::<lambda(int)>, void, int>::invoke(boost::detail::function::function_buffer &, int) (function_obj_ptr=..., a0=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:159 #19 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768 #20 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522 #21 0x000055712c3332d9 in Context::complete (this=0x55712eb856e0, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77 #22 0x00007f824f3d36ce in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0 #23 0x00007f824e6dc6ba in start_thread (arg=0x7f8235e08700) at pthread_create.c:333 #24 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 32 (Thread 0x7f8236609700 (LWP 2488016)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4f40) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba4f40, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4f40, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f8236609700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 31 (Thread 0x7f8236e0a700 (LWP 2488015)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba6ee0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba6ee0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba6ee0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 ---Type <return> to continue, or q <return> to quit--- #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f8236e0a700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 30 (Thread 0x7f823760b700 (LWP 2488014)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712e6de4a0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712e6de4a0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712e6de4a0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f823760b700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 29 (Thread 0x7f8237e0c700 (LWP 2488013)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba5cc0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba5cc0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba5cc0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f8237e0c700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 28 (Thread 0x7f823860d700 (LWP 2488012)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba5c00) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba5c00, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba5c00, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f823860d700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 27 (Thread 0x7f8238e0e700 (LWP 2488011)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4ec0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba4ec0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4ec0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 ---Type <return> to continue, or q <return> to quit--- #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f8238e0e700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 26 (Thread 0x7f823960f700 (LWP 2488010)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba61c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba61c0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba61c0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f823960f700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 25 (Thread 0x7f8239e10700 (LWP 2488009)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4860) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba4860, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4860, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f8239e10700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 24 (Thread 0x7f823a611700 (LWP 2488008)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4280) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba4280, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4280, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f823a611700) at pthread_create.c:333 ---Type <return> to continue, or q <return> to quit--- #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 23 (Thread 0x7f823ae12700 (LWP 2488007)): #0 0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba6d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x55712eba6d60, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba6d60, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #20 0x00007f824e6dc6ba in start_thread (arg=0x7f823ae12700) at pthread_create.c:333 #21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 22 (Thread 0x7f823b613700 (LWP 2488006)): #0 0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824ebc1316 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #2 0x00007f824ec1583c in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #3 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #4 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824eca146d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ec0e14c in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #14 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #15 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #16 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #17 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #18 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #19 0x00007f824e6dc6ba in start_thread (arg=0x7f823b613700) at pthread_create.c:333 #20 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 21 (Thread 0x7f823be14700 (LWP 2488005)): #0 0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #2 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #3 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #4 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824e6dc6ba in start_thread (arg=0x7f823be14700) at pthread_create.c:333 #13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 20 (Thread 0x7f823c615700 (LWP 2488004)): #0 0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #2 0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #3 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #4 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #5 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #6 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #7 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #8 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #9 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 #13 0x000055712c370d82 in PyModuleRunner::serve (this=0x55712d3a2fa0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:47 #14 0x000055712c3711e7 in PyModuleRunner::PyModuleRunnerThread::entry (this=0x55712d3a2fc8) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:106 #15 0x00007f824e6dc6ba in start_thread (arg=0x7f823c615700) at pthread_create.c:333 #16 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 19 (Thread 0x7f8241a29700 (LWP 2487992)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 ---Type <return> to continue, or q <return> to quit--- #1 0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84559f8) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0 #3 0x000055712c381b5d in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115 #4 PyModuleRegistry::get_modules[abi:cxx11]() const (this=0x7ffdf84559d0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.h:70 #5 MgrStandby::send_beacon (this=this@entry=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:185 #6 0x000055712c382b3f in MgrStandby::tick (this=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:239 #7 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768 #8 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522 #9 0x000055712c3332d9 in Context::complete (this=0x55712eb84570, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77 #10 0x00007f824f3d1aab in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0 #11 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #12 0x00007f824e6dc6ba in start_thread (arg=0x7f8241a29700) at pthread_create.c:333 #13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 18 (Thread 0x7f824222a700 (LWP 2487991)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055712c48d91b in Cond::WaitInterval<std::chrono::duration<long, std::ratio<1l, 1l> > > (interval=..., mutex=..., this=0x55712d39f3f8) at /build/ceph-13.1.0-119-g1df1443/src/common/Cond.h:84 #2 ObjectCacher::flusher_entry (this=0x55712d39f200) at /build/ceph-13.1.0-119-g1df1443/src/osdc/ObjectCacher.cc:1952 #3 0x000055712c4a0e5d in ObjectCacher::FlusherThread::entry (this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/osdc/ObjectCacher.h:432 #4 0x00007f824e6dc6ba in start_thread (arg=0x7f824222a700) at pthread_create.c:333 #5 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 17 (Thread 0x7f8242a2b700 (LWP 2487990)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3d1c09 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f8242a2b700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 16 (Thread 0x7f824322c700 (LWP 2487989)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3d1c09 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f824322c700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 15 (Thread 0x7f8243a2d700 (LWP 2487988)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824e6dc6ba in start_thread (arg=0x7f8243a2d700) at pthread_create.c:333 #3 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 14 (Thread 0x7f824422e700 (LWP 2487987)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00007f824f3d1e45 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f824422e700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 13 (Thread 0x7f8244a2f700 (LWP 2487986)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f44fa10 in DispatchQueue::run_local_delivery() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f4ec10d in DispatchQueue::LocalDeliveryThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f8244a2f700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 12 (Thread 0x7f8245230700 (LWP 2487985)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84558b8) at ../nptl/pthread_mutex_lock.c:80 #2 0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0 #3 0x000055712c381675 in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115 #4 MgrStandby::ms_dispatch (this=0x7ffdf8453770, m=0x55712f620a00) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:411 #5 0x00007f824f44de02 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0 #6 0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #7 0x00007f824e6dc6ba in start_thread (arg=0x7f8245230700) at pthread_create.c:333 #8 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 11 (Thread 0x7f8245a31700 (LWP 2487984)): #0 0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824df170be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27 #2 0x000055712c47270e in poll (__timeout=-1, __nfds=4, __fds=0x7f8245a2e010) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #3 SignalHandler::entry (this=0x55712d53a1c0) at /build/ceph-13.1.0-119-g1df1443/src/global/signal_handler.cc:301 #4 0x00007f824e6dc6ba in start_thread (arg=0x7f8245a31700) at pthread_create.c:333 #5 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 10 (Thread 0x7f8246232700 (LWP 2487983)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824e6dc6ba in start_thread (arg=0x7f8246232700) at pthread_create.c:333 #3 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 9 (Thread 0x7f82473e3700 (LWP 2487982)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824e6dc6ba in start_thread (arg=0x7f82473e3700) at pthread_create.c:333 #3 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ---Type <return> to continue, or q <return> to quit--- Thread 8 (Thread 0x7f8247be4700 (LWP 2487981)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x000055712c387c8a in __gthread_cond_timedwait (__abs_timeout=0x7f8247be1260, __mutex=<optimized out>, __cond=0x7ffdf8453f78) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:871 #2 std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7ffdf8453f78) at /usr/include/c++/7/condition_variable:166 #3 std::condition_variable::wait_until<ceph::time_detail::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7ffdf8453f78) at /usr/include/c++/7/condition_variable:119 #4 ceph::timer_detail::timer<ceph::time_detail::coarse_mono_clock>::timer_thread (this=0x7ffdf8453f10) at /build/ceph-13.1.0-119-g1df1443/src/common/ceph_timer.h:144 #5 0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0 #6 0x00007f824e6dc6ba in start_thread (arg=0x7f8247be4700) at pthread_create.c:333 #7 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 7 (Thread 0x7f82483e5700 (LWP 2487980)): #0 0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824f3b3804 in AdminSocket::entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824e6dc6ba in start_thread (arg=0x7f82483e5700) at pthread_create.c:333 #4 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 6 (Thread 0x7f8248be6700 (LWP 2487979)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00007f824f5d5906 in CephContextServiceThread::entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824e6dc6ba in start_thread (arg=0x7f8248be6700) at pthread_create.c:333 #3 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 5 (Thread 0x7f82493e7700 (LWP 2487974)): #0 0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0 #4 0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0 #5 0x00007f824e6dc6ba in start_thread (arg=0x7f82493e7700) at pthread_create.c:333 #6 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 4 (Thread 0x7f8249be8700 (LWP 2487973)): #0 0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0 #4 0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0 #5 0x00007f824e6dc6ba in start_thread (arg=0x7f8249be8700) at pthread_create.c:333 #6 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 3 (Thread 0x7f824a3e9700 (LWP 2487972)): #0 0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0 #3 0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0 #4 0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0 #5 0x00007f824e6dc6ba in start_thread (arg=0x7f824a3e9700) at pthread_create.c:333 #6 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7f824b388700 (LWP 2487966)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f3f7a8d in ceph::logging::Log::entry() () from /usr/lib/ceph/libceph-common.so.0 #2 0x00007f824e6dc6ba in start_thread (arg=0x7f824b388700) at pthread_create.c:333 #3 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7f825809b380 (LWP 2487960)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f824f50c657 in AsyncMessenger::wait() () from /usr/lib/ceph/libceph-common.so.0 #2 0x000055712c2c887d in MgrStandby::main (this=0x7ffdf8453770, args=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:470 #3 0x000055712c2c6d3f in main (argc=10, argv=0x7ffdf8455d88) at /build/ceph-13.1.0-119-g1df1443/src/ceph_mgr.cc:77
#2 Updated by John Spray almost 6 years ago
Because the StandbyPyModules::shutdown stack is stuck in python execution, it makes me think this is happening when one of the python modules is being naughty and blocking somehow in shutdown() -- the dashboard module blocks waiting for its server to start, which is fixed by https://github.com/ceph/ceph/pull/22051, hopefully that PR fixes this case.
Logs with "debug mgr = 10" would confirm which module is offending.
#3 Updated by Sage Weil almost 6 years ago
- Priority changed from Immediate to Urgent
downgrading since this is a shutdown bug...
#4 Updated by Sage Weil about 5 years ago
- Status changed from 12 to Can't reproduce