Project

General

Profile

Bug #24166

mgr: deadlock between PyModuleRegistry and ms_dispatch and ...

Added by Sage Weil almost 6 years ago. Updated about 5 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84558b8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381675 in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  MgrStandby::ms_dispatch (this=0x7ffdf8453770, m=0x55712f620a00) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:411
#5  0x00007f824f44de02 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0
#6  0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#7  0x00007f824e6dc6ba in start_thread (arg=0x7f8245230700) at pthread_create.c:333
#8  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84559f8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381b5d in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  PyModuleRegistry::get_modules[abi:cxx11]() const (this=0x7ffdf84559d0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.h:70
#5  MgrStandby::send_beacon (this=this@entry=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:185
#6  0x000055712c382b3f in MgrStandby::tick (this=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:239
#7  0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#8  FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#9  0x000055712c3332d9 in Context::complete (this=0x55712eb84570, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#10 0x00007f824f3d1aab in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#11 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#12 0x00007f824e6dc6ba in start_thread (arg=0x7f8241a29700) at pthread_create.c:333
#13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 33 (Thread 0x7f8235e08700 (LWP 2489629)):
#0  0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x000055712c371258 in PyModuleRunner::shutdown (this=this@entry=0x55712d3a2f40) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:82
#14 0x000055712c3616ac in StandbyPyModules::shutdown (this=0x55712d3dd8c0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/StandbyPyModules.cc:56
#15 0x000055712c36c623 in PyModuleRegistry::active_start (this=0x7ffdf84559d0, ds=..., cs=..., kv_store=std::map with 5 elements = {...}, mc=..., clog_=std::shared_ptr (count 9, weak 0) 0x55712d41b7b0, objecter_=..., client_=..., 
    f=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.cc:187
#16 0x000055712c3946b7 in Mgr::init (this=0x55712ed52000) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:263
#17 0x000055712c394a1c in Mgr::<lambda(int)>::operator() (r=<optimized out>, __closure=0x55712eb856f0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:148
#18 boost::detail::function::void_function_obj_invoker1<Mgr::background_init(Context*)::<lambda(int)>, void, int>::invoke(boost::detail::function::function_buffer &, int) (function_obj_ptr=..., a0=<optimized out>)
    at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:159
#19 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#20 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#21 0x000055712c3332d9 in Context::complete (this=0x55712eb856e0, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#22 0x00007f824f3d36ce in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#23 0x00007f824e6dc6ba in start_thread (arg=0x7f8235e08700) at pthread_create.c:333
#24 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

on lab cluster

History

#1 Updated by Sage Weil almost 6 years ago

all threads

(gdb) thr app all bt

Thread 35 (Thread 0x7f8234e06700 (LWP 2489631)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f44fa10 in DispatchQueue::run_local_delivery() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f4ec10d in DispatchQueue::LocalDeliveryThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f8234e06700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 34 (Thread 0x7f8235607700 (LWP 2489630)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f44d907 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f8235607700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 33 (Thread 0x7f8235e08700 (LWP 2489629)):
#0  0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x000055712c371258 in PyModuleRunner::shutdown (this=this@entry=0x55712d3a2f40) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:82
#14 0x000055712c3616ac in StandbyPyModules::shutdown (this=0x55712d3dd8c0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/StandbyPyModules.cc:56
#15 0x000055712c36c623 in PyModuleRegistry::active_start (this=0x7ffdf84559d0, ds=..., cs=..., kv_store=std::map with 5 elements = {...}, mc=..., clog_=std::shared_ptr (count 9, weak 0) 0x55712d41b7b0, objecter_=..., client_=..., 
    f=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.cc:187
#16 0x000055712c3946b7 in Mgr::init (this=0x55712ed52000) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:263
#17 0x000055712c394a1c in Mgr::<lambda(int)>::operator() (r=<optimized out>, __closure=0x55712eb856f0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/Mgr.cc:148
#18 boost::detail::function::void_function_obj_invoker1<Mgr::background_init(Context*)::<lambda(int)>, void, int>::invoke(boost::detail::function::function_buffer &, int) (function_obj_ptr=..., a0=<optimized out>)
    at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:159
#19 0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#20 FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#21 0x000055712c3332d9 in Context::complete (this=0x55712eb856e0, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#22 0x00007f824f3d36ce in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#23 0x00007f824e6dc6ba in start_thread (arg=0x7f8235e08700) at pthread_create.c:333
#24 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 32 (Thread 0x7f8236609700 (LWP 2488016)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4f40) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba4f40, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4f40, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f8236609700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 31 (Thread 0x7f8236e0a700 (LWP 2488015)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba6ee0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba6ee0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba6ee0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
---Type <return> to continue, or q <return> to quit---
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f8236e0a700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 30 (Thread 0x7f823760b700 (LWP 2488014)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712e6de4a0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712e6de4a0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712e6de4a0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f823760b700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 29 (Thread 0x7f8237e0c700 (LWP 2488013)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba5cc0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba5cc0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba5cc0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f8237e0c700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 28 (Thread 0x7f823860d700 (LWP 2488012)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba5c00) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba5c00, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba5c00, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f823860d700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 27 (Thread 0x7f8238e0e700 (LWP 2488011)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4ec0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba4ec0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4ec0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
---Type <return> to continue, or q <return> to quit---
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f8238e0e700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 26 (Thread 0x7f823960f700 (LWP 2488010)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba61c0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba61c0, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba61c0, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f823960f700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 25 (Thread 0x7f8239e10700 (LWP 2488009)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4860) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba4860, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4860, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f8239e10700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 24 (Thread 0x7f823a611700 (LWP 2488008)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba4280) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba4280, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba4280, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f823a611700) at pthread_create.c:333
---Type <return> to continue, or q <return> to quit---
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 23 (Thread 0x7f823ae12700 (LWP 2488007)):
#0  0x00007f824e6e4827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x55712eba6d60) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x55712eba6d60, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f824e6e48d4 in __new_sem_wait_slow (sem=0x55712eba6d60, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f824e6e497a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00007f824ec37fe8 in PyThread_acquire_lock () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec69832 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#20 0x00007f824e6dc6ba in start_thread (arg=0x7f823ae12700) at pthread_create.c:333
#21 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 22 (Thread 0x7f823b613700 (LWP 2488006)):
#0  0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ebc1316 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec1583c in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824eca146d in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec0e14c in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#14 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#15 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#16 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#17 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#18 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#19 0x00007f824e6dc6ba in start_thread (arg=0x7f823b613700) at pthread_create.c:333
#20 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 21 (Thread 0x7f823be14700 (LWP 2488005)):
#0  0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec14044 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ed4a487 in PyEval_CallObjectWithKeywords () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec6d0b2 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824e6dc6ba in start_thread (arg=0x7f823be14700) at pthread_create.c:333
#13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 20 (Thread 0x7f823c615700 (LWP 2488004)):
#0  0x00007f824defb5d3 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824ec6c990 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#2  0x00007f824ec15971 in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#3  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#4  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#5  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#6  0x00007f824ec13f1d in PyEval_EvalFrameEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#7  0x00007f824ed4b05c in PyEval_EvalCodeEx () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#8  0x00007f824eca1370 in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#9  0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#10 0x00007f824ece83ac in ?? () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#11 0x00007f824ec74273 in PyObject_Call () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#12 0x00007f824ec75444 in PyObject_CallMethod () from /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
#13 0x000055712c370d82 in PyModuleRunner::serve (this=0x55712d3a2fa0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:47
#14 0x000055712c3711e7 in PyModuleRunner::PyModuleRunnerThread::entry (this=0x55712d3a2fc8) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRunner.cc:106
#15 0x00007f824e6dc6ba in start_thread (arg=0x7f823c615700) at pthread_create.c:333
#16 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 19 (Thread 0x7f8241a29700 (LWP 2487992)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
---Type <return> to continue, or q <return> to quit---
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84559f8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381b5d in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  PyModuleRegistry::get_modules[abi:cxx11]() const (this=0x7ffdf84559d0) at /build/ceph-13.1.0-119-g1df1443/src/mgr/PyModuleRegistry.h:70
#5  MgrStandby::send_beacon (this=this@entry=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:185
#6  0x000055712c382b3f in MgrStandby::tick (this=0x7ffdf8453770) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:239
#7  0x000055712c337c5c in boost::function1<void, int>::operator() (a0=<optimized out>, this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/obj-x86_64-linux-gnu/boost/include/boost/function/function_template.hpp:768
#8  FunctionContext::finish (this=<optimized out>, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:522
#9  0x000055712c3332d9 in Context::complete (this=0x55712eb84570, r=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/include/Context.h:77
#10 0x00007f824f3d1aab in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#11 0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#12 0x00007f824e6dc6ba in start_thread (arg=0x7f8241a29700) at pthread_create.c:333
#13 0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 18 (Thread 0x7f824222a700 (LWP 2487991)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055712c48d91b in Cond::WaitInterval<std::chrono::duration<long, std::ratio<1l, 1l> > > (interval=..., mutex=..., this=0x55712d39f3f8) at /build/ceph-13.1.0-119-g1df1443/src/common/Cond.h:84
#2  ObjectCacher::flusher_entry (this=0x55712d39f200) at /build/ceph-13.1.0-119-g1df1443/src/osdc/ObjectCacher.cc:1952
#3  0x000055712c4a0e5d in ObjectCacher::FlusherThread::entry (this=<optimized out>) at /build/ceph-13.1.0-119-g1df1443/src/osdc/ObjectCacher.h:432
#4  0x00007f824e6dc6ba in start_thread (arg=0x7f824222a700) at pthread_create.c:333
#5  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 17 (Thread 0x7f8242a2b700 (LWP 2487990)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3d1c09 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f8242a2b700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 16 (Thread 0x7f824322c700 (LWP 2487989)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3d1c09 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f824322c700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 15 (Thread 0x7f8243a2d700 (LWP 2487988)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824e6dc6ba in start_thread (arg=0x7f8243a2d700) at pthread_create.c:333
#3  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 14 (Thread 0x7f824422e700 (LWP 2487987)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f824f3d1e45 in SafeTimer::timer_thread() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f3d306d in SafeTimerThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f824422e700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 13 (Thread 0x7f8244a2f700 (LWP 2487986)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f44fa10 in DispatchQueue::run_local_delivery() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f4ec10d in DispatchQueue::LocalDeliveryThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f8244a2f700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 12 (Thread 0x7f8245230700 (LWP 2487985)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f824e6dedbd in __GI___pthread_mutex_lock (mutex=0x7ffdf84558b8) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f824f3a9f49 in Mutex::Lock(bool) () from /usr/lib/ceph/libceph-common.so.0
#3  0x000055712c381675 in Mutex::Locker::Locker (m=..., this=<synthetic pointer>) at /build/ceph-13.1.0-119-g1df1443/src/common/Mutex.h:115
#4  MgrStandby::ms_dispatch (this=0x7ffdf8453770, m=0x55712f620a00) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:411
#5  0x00007f824f44de02 in DispatchQueue::entry() () from /usr/lib/ceph/libceph-common.so.0
#6  0x00007f824f4ebffd in DispatchQueue::DispatchThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#7  0x00007f824e6dc6ba in start_thread (arg=0x7f8245230700) at pthread_create.c:333
#8  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 11 (Thread 0x7f8245a31700 (LWP 2487984)):
#0  0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824df170be in __poll_chk (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>, fdslen=<optimized out>) at poll_chk.c:27
#2  0x000055712c47270e in poll (__timeout=-1, __nfds=4, __fds=0x7f8245a2e010) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
#3  SignalHandler::entry (this=0x55712d53a1c0) at /build/ceph-13.1.0-119-g1df1443/src/global/signal_handler.cc:301
#4  0x00007f824e6dc6ba in start_thread (arg=0x7f8245a31700) at pthread_create.c:333
#5  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 10 (Thread 0x7f8246232700 (LWP 2487983)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824e6dc6ba in start_thread (arg=0x7f8246232700) at pthread_create.c:333
#3  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 9 (Thread 0x7f82473e3700 (LWP 2487982)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3d3878 in Finisher::finisher_thread_entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824e6dc6ba in start_thread (arg=0x7f82473e3700) at pthread_create.c:333
#3  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
---Type <return> to continue, or q <return> to quit---

Thread 8 (Thread 0x7f8247be4700 (LWP 2487981)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055712c387c8a in __gthread_cond_timedwait (__abs_timeout=0x7f8247be1260, __mutex=<optimized out>, __cond=0x7ffdf8453f78) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:871
#2  std::condition_variable::__wait_until_impl<std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=<synthetic pointer>, __lock=..., this=0x7ffdf8453f78) at /usr/include/c++/7/condition_variable:166
#3  std::condition_variable::wait_until<ceph::time_detail::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x7ffdf8453f78)
    at /usr/include/c++/7/condition_variable:119
#4  ceph::timer_detail::timer<ceph::time_detail::coarse_mono_clock>::timer_thread (this=0x7ffdf8453f10) at /build/ceph-13.1.0-119-g1df1443/src/common/ceph_timer.h:144
#5  0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0
#6  0x00007f824e6dc6ba in start_thread (arg=0x7f8247be4700) at pthread_create.c:333
#7  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f82483e5700 (LWP 2487980)):
#0  0x00007f824def974d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824f3b3804 in AdminSocket::entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824e6dc6ba in start_thread (arg=0x7f82483e5700) at pthread_create.c:333
#4  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f8248be6700 (LWP 2487979)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x00007f824f5d5906 in CephContextServiceThread::entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824e6dc6ba in start_thread (arg=0x7f8248be6700) at pthread_create.c:333
#3  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7f82493e7700 (LWP 2487974)):
#0  0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0
#4  0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0
#5  0x00007f824e6dc6ba in start_thread (arg=0x7f82493e7700) at pthread_create.c:333
#6  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f8249be8700 (LWP 2487973)):
#0  0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0
#4  0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0
#5  0x00007f824e6dc6ba in start_thread (arg=0x7f8249be8700) at pthread_create.c:333
#6  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f824a3e9700 (LWP 2487972)):
#0  0x00007f824df05a13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f824f77acf9 in EpollDriver::event_wait(std::vector<FiredFileEvent, std::allocator<FiredFileEvent> >&, timeval*) () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824f50f4e7 in EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*) () from /usr/lib/ceph/libceph-common.so.0
#3  0x00007f824f51275c in ?? () from /usr/lib/ceph/libceph-common.so.0
#4  0x00007f824f8968ff in ?? () from /usr/lib/ceph/libceph-common.so.0
#5  0x00007f824e6dc6ba in start_thread (arg=0x7f824a3e9700) at pthread_create.c:333
#6  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f824b388700 (LWP 2487966)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f3f7a8d in ceph::logging::Log::entry() () from /usr/lib/ceph/libceph-common.so.0
#2  0x00007f824e6dc6ba in start_thread (arg=0x7f824b388700) at pthread_create.c:333
#3  0x00007f824df0541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f825809b380 (LWP 2487960)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f824f50c657 in AsyncMessenger::wait() () from /usr/lib/ceph/libceph-common.so.0
#2  0x000055712c2c887d in MgrStandby::main (this=0x7ffdf8453770, args=...) at /build/ceph-13.1.0-119-g1df1443/src/mgr/MgrStandby.cc:470
#3  0x000055712c2c6d3f in main (argc=10, argv=0x7ffdf8455d88) at /build/ceph-13.1.0-119-g1df1443/src/ceph_mgr.cc:77

#2 Updated by John Spray almost 6 years ago

Because the StandbyPyModules::shutdown stack is stuck in python execution, it makes me think this is happening when one of the python modules is being naughty and blocking somehow in shutdown() -- the dashboard module blocks waiting for its server to start, which is fixed by https://github.com/ceph/ceph/pull/22051, hopefully that PR fixes this case.

Logs with "debug mgr = 10" would confirm which module is offending.

#3 Updated by Sage Weil almost 6 years ago

  • Priority changed from Immediate to Urgent

downgrading since this is a shutdown bug...

#4 Updated by Sage Weil about 5 years ago

  • Status changed from 12 to Can't reproduce

Also available in: Atom PDF