Actions
Bug #9182
closedosd deadlock after ms_handle_reset
Added by Sage Weil over 9 years ago. Updated about 9 years ago.
Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
ubuntu@teuthology:/a/teuthology-2014-08-19_02:30:02-rados-firefly-distro-basic-multi/435572
2014-08-20 10:50:59.430375 7f67deb9c700 1 osd.0 515 ms_handle_reset con 0x351e420 session 0x3f92a80
is the last thing from teh dispatch thread. gdb says
(gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fdd50) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fdce8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67deb9c700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? ()
Updated by Sage Weil over 9 years ago
(gdb) thr app all bt Thread 119 (Thread 0x7f67ed290700 (LWP 31542)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009c57fb in ceph::log::Log::entry (this=0x31e5000) at log/Log.cc:323 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67ed290700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 118 (Thread 0x7f67ec341700 (LWP 31543)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a88c46 in WaitUntil (when=..., mutex=..., this=0x31f14e8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31f14e8) at ./common/Cond.h:79 #3 CephContextServiceThread::entry (this=0x31f1480) at common/ceph_context.cc:58 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67ec341700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 117 (Thread 0x7f67ebb40700 (LWP 31544)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000a6e8bb in AdminSocket::entry (this=0x31e5140) at common/admin_socket.cc:239 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67ebb40700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 116 (Thread 0x7f67eb33f700 (LWP 31552)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f67efae6f33 in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67eb33f700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 115 (Thread 0x7f67e61e0700 (LWP 31563)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x31fd518) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x31fd000) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e61e0700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 114 (Thread 0x7f67e69e1700 (LWP 31564)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x31fdb18) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x31fd600) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e69e1700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 113 (Thread 0x7f67e71e2700 (LWP 31565)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x31fe718) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x31fe200) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e71e2700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 112 (Thread 0x7f67e91e6700 (LWP 31566)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x31fed18) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x31fe800) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e91e6700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 111 (Thread 0x7f67eab3e700 (LWP 31567)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x31fe118) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x31fdc00) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67eab3e700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 110 (Thread 0x7f67ea33d700 (LWP 31568)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a52f69 in Wait (mutex=..., this=0x3201718) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x3201200) at msg/SimpleMessenger.cc:210 #3 0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67ea33d700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 109 (Thread 0x7f67e99e7700 (LWP 31569)): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x36bf080) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x36bf070) at common/Timer.cc:114 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e99e7700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 108 (Thread 0x7f67e89e5700 (LWP 31570)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a62187 in Wait (mutex=..., this=0x36c1520) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x36c1510) at common/Timer.cc:112 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e89e5700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 107 (Thread 0x7f67e81e4700 (LWP 31571)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009053fc in Wait (mutex=..., this=0x31eeaa0) at ./common/Cond.h:55 #2 WBThrottle::get_next_should_flush (this=0x31ee998, next=0x7f67e81e3d00) at os/WBThrottle.cc:142 #3 0x0000000000905ca4 in WBThrottle::entry (this=0x31ee998) at os/WBThrottle.cc:160 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e81e4700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 106 (Thread 0x7f67e79e3700 (LWP 31572)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x000000000088b638 in WaitUntil (when=..., mutex=..., this=0x31ee680) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31ee680) at ./common/Cond.h:79 #3 FileStore::sync_entry (this=0x31ee000) at os/FileStore.cc:3097 #4 0x00000000008a98dd in FileStore::SyncThread::entry (this=<optimized out>) at os/FileStore.h:178 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67e79e3700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 105 (Thread 0x7f67e59df700 (LWP 31573)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000971e0d in Wait (mutex=..., this=0x31e8138) at ./common/Cond.h:55 #2 FileJournal::write_thread_entry (this=0x31e8000) at os/FileJournal.cc:1109 #3 0x00000000008a324d in FileJournal::Writer::entry (this=<optimized out>) at os/FileJournal.h:334 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e59df700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 104 (Thread 0x7f67e51de700 (LWP 31574)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009697d9 in Wait (mutex=..., this=0x31e8380) at ./common/Cond.h:55 #2 FileJournal::write_finish_thread_entry (this=0x31e8000) at os/FileJournal.cc:1333 #3 0x00000000008a322d in FileJournal::WriteFinisher::entry (this=<optimized out>) at os/FileJournal.h:344 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67e51de700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 103 (Thread 0x7f67e49dd700 (LWP 31575)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x31ee080) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x31ee020) at common/Finisher.cc:78 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67e49dd700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 102 (Thread 0x7f67e41dc700 (LWP 31576)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x31eee08) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31eee08) at common/Cond.h:79 #3 ThreadPool::worker (this=0x31eed90, wt=0x3560080) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67e41dc700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 101 (Thread 0x7f67e39db700 (LWP 31577)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x31eee08) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31eee08) at common/Cond.h:79 #3 ThreadPool::worker (this=0x31eed90, wt=0x35602c0) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67e39db700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 100 (Thread 0x7f67e31da700 (LWP 31578)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x31eecc0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x31eec60) at common/Finisher.cc:78 ---Type <return> to continue, or q <return> to quit--- #3 0x00007f67ef62be9a in start_thread (arg=0x7f67e31da700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 99 (Thread 0x7f67e29d9700 (LWP 31579)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x31ee550) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x31ee4f0) at common/Finisher.cc:78 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67e29d9700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 98 (Thread 0x7f67dfb9e700 (LWP 31581)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fd150) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fd0e8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67dfb9e700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 97 (Thread 0x7f67df39d700 (LWP 31582)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b0b3b4 in Accepter::entry (this=0x31fd0c0) at msg/Accepter.cc:196 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67df39d700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 96 (Thread 0x7f67deb9c700 (LWP 31583)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fdd50) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fdce8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67deb9c700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 95 (Thread 0x7f67ddb9a700 (LWP 31585)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fd750) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fd6e8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67ddb9a700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 94 (Thread 0x7f67dd399700 (LWP 31586)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fe350) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fe2e8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67dd399700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 93 (Thread 0x7f67dc397700 (LWP 31588)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fe950) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x31fe8e8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67dc397700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 92 (Thread 0x7f67db395700 (LWP 31590)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b1d3c8 in Wait (mutex=..., this=0x3201350) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x32012e8) at msg/DispatchQueue.cc:138 #3 0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67db395700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 91 (Thread 0x7f67dab94700 (LWP 31591)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b0b3b4 in Accepter::entry (this=0x32012c0) at msg/Accepter.cc:196 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67dab94700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 90 (Thread 0x7f67da393700 (LWP 31592)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x7fff640b45d8) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x7fff640b45c8) at common/Timer.cc:114 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67da393700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 89 (Thread 0x7f67d9b92700 (LWP 31593)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x7fff640b46f0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7fff640b4690) at common/Finisher.cc:78 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d9b92700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 88 (Thread 0x7f67d9391700 (LWP 31594)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf4e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf4e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x36bf470, wt=0x43d3720) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d9391700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 87 (Thread 0x7f67d8b90700 (LWP 31595)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf4e8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf4e8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x36bf470, wt=0x4391d80) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d8b90700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 86 (Thread 0x7f67d838f700 (LWP 31596)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf670) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf670) at common/Cond.h:79 #3 ThreadPool::worker (this=0x36bf5f8, wt=0x43d3600) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d838f700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 85 (Thread 0x7f67d7b8e700 (LWP 31597)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf7f8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf7f8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x36bf780, wt=0x43d36a0) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d7b8e700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 84 (Thread 0x7f67d738d700 (LWP 31598)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf980) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf980) at common/Cond.h:79 #3 ThreadPool::worker (this=0x36bf908, wt=0x43d3400) at common/WorkQueue.cc:143 #4 0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d738d700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 83 (Thread 0x7f67d6b8c700 (LWP 31599)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x000000000063507b in WaitUntil (when=..., mutex=..., this=0x36bfb20) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bfb20) at ./common/Cond.h:79 #3 OSD::heartbeat_entry (this=0x36bf000) at osd/OSD.cc:3084 #4 0x000000000068e87d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:1041 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d6b8c700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 82 (Thread 0x7f67d638b700 (LWP 31600)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x36c1698) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x36c1638) at common/Finisher.cc:78 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d638b700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 81 (Thread 0x7f67d5b8a700 (LWP 31601)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009b5c80 in Wait (mutex=..., this=0x36c12a8) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x36c1248) at common/Finisher.cc:78 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d5b8a700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 80 (Thread 0x7f67d5389700 (LWP 31602)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x36c0f88) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x36c0f78) at common/Timer.cc:114 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d5389700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 79 (Thread 0x7f67d4b88700 (LWP 31604)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a62187 in Wait (mutex=..., this=0x36c13f8) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x36c13e8) at common/Timer.cc:112 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d4b88700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 78 (Thread 0x7f67d4387700 (LWP 31605)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000a62187 in Wait (mutex=..., this=0x36c0e68) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x36c0e58) at common/Timer.cc:112 #3 0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d4387700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 77 (Thread 0x7f67d3b86700 (LWP 31606)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000615664 in Wait (mutex=..., this=<optimized out>) at ./common/Cond.h:55 #2 OSDService::agent_entry (this=0x36c06b8) at osd/OSD.cc:514 #3 0x0000000000673a2d in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:465 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d3b86700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 76 (Thread 0x7f67d0b79700 (LWP 31623)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x00000000009951e4 in SignalHandler::entry (this=0x3dc4780) at global/signal_handler.cc:213 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67d0b79700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 75 (Thread 0x7f67bf96d700 (LWP 12970)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b0b3b4 in Accepter::entry (this=0x31fdcc0) at msg/Accepter.cc:196 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67bf96d700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 74 (Thread 0x7f67dbb96700 (LWP 12971)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b0b3b4 in Accepter::entry (this=0x31fe8c0) at msg/Accepter.cc:196 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67dbb96700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 73 (Thread 0x7f67dcb98700 (LWP 12972)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b0b3b4 in Accepter::entry (this=0x31fe2c0) at msg/Accepter.cc:196 #2 0x00007f67ef62be9a in start_thread (arg=0x7f67dcb98700) at pthread_create.c:308 #3 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 72 (Thread 0x7f67c3c3d700 (LWP 12982)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x48f5970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x48f5780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c3c3d700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 71 (Thread 0x7f67ce568700 (LWP 12983)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3b21e70) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3b21c80) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67ce568700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 70 (Thread 0x7f67be062700 (LWP 12984)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x464c1f0) at ./common/Cond.h:55 ---Type <return> to continue, or q <return> to quit--- #2 Pipe::writer (this=0x464c000) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67be062700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 69 (Thread 0x7f67cfc71700 (LWP 12985)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x464cbf0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x464ca00) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67cfc71700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 68 (Thread 0x7f67c854e700 (LWP 12987)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3926470) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3926280) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c854e700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 67 (Thread 0x7f67be567700 (LWP 12988)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3926970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3926780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67be567700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 66 (Thread 0x7f67de29a700 (LWP 12992)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x526abf0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x526aa00) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67de29a700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 65 (Thread 0x7f67de199700 (LWP 12995)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x526a470) at ./common/Cond.h:55 #2 Pipe::writer (this=0x526a280) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67de199700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 64 (Thread 0x7f67ddf97700 (LWP 13005)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x48f5780, buf=0x7f67ddf96e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x48f5780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67ddf97700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 63 (Thread 0x7f67ca855700 (LWP 13006)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x3f54540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3f54480) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67ca855700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 62 (Thread 0x7f67dde96700 (LWP 13007)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21c80, buf=0x7f67dde95e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3b21c80) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67dde96700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 61 (Thread 0x7f67cd263700 (LWP 13008)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x3f540c0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3f54000) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67cd263700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 60 (Thread 0x7f67ddd95700 (LWP 13009)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x464ca00, buf=0x7f67ddd94e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x464ca00) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67ddd95700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 59 (Thread 0x7f67d3284700 (LWP 13010)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3926280, buf=0x7f67d3283e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3926280) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d3284700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 58 (Thread 0x7f67ce467700 (LWP 13011)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x3f54c00) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3f54b40) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67ce467700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 57 (Thread 0x7f67cf76c700 (LWP 13012)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x3f54420) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3f54360) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67cf76c700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 56 (Thread 0x7f67d2781700 (LWP 13013)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x464c000, buf=0x7f67d2780e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x464c000) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d2781700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 55 (Thread 0x7f67d237c700 (LWP 13014)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3926780, buf=0x7f67d237be1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3926780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d237c700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 54 (Thread 0x7f67d1d7d700 (LWP 13015)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x444a540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x444a480) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d1d7d700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 53 (Thread 0x7f67d137a700 (LWP 13016)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x444a300) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x444a240) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d137a700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 52 (Thread 0x7f67d0378700 (LWP 13020)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x526aa00, buf=0x7f67d0377e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x526aa00) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d0378700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 51 (Thread 0x7f67d0176700 (LWP 13019)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 ---Type <return> to continue, or q <return> to quit--- #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x526a280, buf=0x7f67d0175e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x526a280) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67d0176700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 50 (Thread 0x7f67cb857700 (LWP 13022)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d71e0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d7120) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67cb857700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 49 (Thread 0x7f67cdb65700 (LWP 13023)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d70c0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d7000) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67cdb65700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 48 (Thread 0x7f67cf86d700 (LWP 13027)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3d291f0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3d29000) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67cf86d700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 47 (Thread 0x7f67ce669700 (LWP 13028)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3d29000, buf=0x7f67ce668e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3d29000) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67ce669700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 46 (Thread 0x7f67c9853700 (LWP 13029)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d7ae0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d7a20) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c9853700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 45 (Thread 0x7f67c3839700 (LWP 13042)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x526ac80, buf=0x7f67c3838e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x526ac80) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c3839700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 44 (Thread 0x7f67bdd5f700 (LWP 13043)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x526a780, buf=0x7f67bdd5ee1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x526a780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67bdd5f700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 43 (Thread 0x7f67c2071700 (LWP 13044)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x5241000, buf=0x7f67c2070e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x5241000) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c2071700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 42 (Thread 0x7f67c52d2700 (LWP 13045)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x4585780, buf=0x7f67c52d1e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x4585780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c52d2700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 41 (Thread 0x7f67c3435700 (LWP 13046)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x5241a00, buf=0x7f67c3434e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x5241a00) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c3435700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 40 (Thread 0x7f67c50d0700 (LWP 13047)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x526ae70) at ./common/Cond.h:55 #2 Pipe::writer (this=0x526ac80) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c50d0700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 39 (Thread 0x7f67c734a700 (LWP 13048)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d7f60) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d7ea0) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c734a700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 38 (Thread 0x7f67c5945700 (LWP 13049)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x526a970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x526a780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c5945700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 37 (Thread 0x7f67c6b49700 (LWP 13050)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d7540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d7480) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c6b49700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 36 (Thread 0x7f67cbb5a700 (LWP 13051)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x5241bf0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x5241a00) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67cbb5a700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 35 (Thread 0x7f67c6348700 (LWP 13052)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d7660) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x35d75a0) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c6348700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 34 (Thread 0x7f67c4fcf700 (LWP 13053)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x4585970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4585780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c4fcf700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 33 (Thread 0x7f67be365700 (LWP 13054)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x52411f0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x5241000) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67be365700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 32 (Thread 0x7f67c4ece700 (LWP 13055)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35d7780) at ./common/Cond.h:55 ---Type <return> to continue, or q <return> to quit--- #2 Pipe::DelayedDelivery::entry (this=0x35d76c0) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c4ece700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 31 (Thread 0x7f67c46cd700 (LWP 13056)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x35610c0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3561000) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c46cd700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 30 (Thread 0x7f67c7f48700 (LWP 13057)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21780, buf=0x7f67c7f47e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3b21780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c7f48700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 29 (Thread 0x7f67c3637700 (LWP 13058)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3b21970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3b21780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67c3637700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 28 (Thread 0x7f67c2bb5700 (LWP 13059)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x3561300) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x3561240) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c2bb5700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 27 (Thread 0x7f67d247d700 (LWP 13062)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3b216f0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3b21500) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d247d700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 26 (Thread 0x7f67c7b4b700 (LWP 13063)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x360d780) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x360d6c0) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c7b4b700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 25 (Thread 0x7f67c0d6c700 (LWP 13069)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21280, buf=0x7f67c0d6be1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x3b21280) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c0d6c700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 24 (Thread 0x7f67bef6a700 (LWP 13070)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x3b21470) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3b21280) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67bef6a700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 23 (Thread 0x7f67d3183700 (LWP 13071)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x49a4420) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x49a4360) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67d3183700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 22 (Thread 0x7f67d217a700 (LWP 13085)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 ---Type <return> to continue, or q <return> to quit--- #1 0x0000000000b3d103 in Wait (mutex=..., this=0x42eb970) at ./common/Cond.h:55 #2 Pipe::writer (this=0x42eb780) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67d217a700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 21 (Thread 0x7f67cca62700 (LWP 13087)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x42eb780, buf=0x7f67cca61e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x42eb780) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67cca62700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 20 (Thread 0x7f67c1a6b700 (LWP 13090)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x360d540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x360d480) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67c1a6b700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 19 (Thread 0x7f67bd059700 (LWP 13280)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x5241470) at ./common/Cond.h:55 #2 Pipe::writer (this=0x5241280) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67bd059700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 18 (Thread 0x7f67bd15a700 (LWP 13284)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x5241280, buf=0x7f67bd159e1a "\377", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x5241280) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67bd15a700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 17 (Thread 0x7f67c3738700 (LWP 13437)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x4f99500, buf=0x7f67c3737e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255 #3 0x0000000000b44620 in Pipe::reader (this=0x4f99500) at msg/Pipe.cc:1421 #4 0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #5 0x00007f67ef62be9a in start_thread (arg=0x7f67c3738700) at pthread_create.c:308 #6 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f67cfa6f700 (LWP 13438)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b3d103 in Wait (mutex=..., this=0x4f996f0) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4f99500) at msg/Pipe.cc:1730 #3 0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00007f67ef62be9a in start_thread (arg=0x7f67cfa6f700) at pthread_create.c:308 #5 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f67ca054700 (LWP 13439)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000b2d099 in Wait (mutex=..., this=0x360d420) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x360d360) at msg/Pipe.cc:209 #3 0x00007f67ef62be9a in start_thread (arg=0x7f67ca054700) at pthread_create.c:308 #4 0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f67c1b6c700 (LWP 13442)): #0 0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282 #2 0x0000000000b30b20 in Pipe::tcp_read (this=0x4585000, buf=0x7f67c1b6be1a "\377", len=1) at ms
Updated by Sage Weil over 9 years ago
..and when i detached gdb the osd saw it was marked down, and came back to life after that. :/
Updated by Samuel Just about 9 years ago
- Status changed from Need More Info to Can't reproduce
Actions