Project

General

Profile

Actions
Copy link

Bug #9182

closed

osd deadlock after ms_handle_reset

Added by Sage Weil over 9 years ago. Updated about 9 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@teuthology:/a/teuthology-2014-08-19_02:30:02-rados-firefly-distro-basic-multi/435572

2014-08-20 10:50:59.430375 7f67deb9c700  1 osd.0 515 ms_handle_reset con 0x351e420 session 0x3f92a80

is the last thing from teh dispatch thread. gdb says
(gdb) bt
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fdd50) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fdce8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67deb9c700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Actions
Copy link
 #1

Updated by Sage Weil over 9 years ago 

(gdb) thr app all bt

Thread 119 (Thread 0x7f67ed290700 (LWP 31542)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009c57fb in ceph::log::Log::entry (this=0x31e5000) at log/Log.cc:323
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67ed290700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 118 (Thread 0x7f67ec341700 (LWP 31543)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a88c46 in WaitUntil (when=..., mutex=..., this=0x31f14e8) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31f14e8) at ./common/Cond.h:79
#3  CephContextServiceThread::entry (this=0x31f1480) at common/ceph_context.cc:58
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67ec341700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 117 (Thread 0x7f67ebb40700 (LWP 31544)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000a6e8bb in AdminSocket::entry (this=0x31e5140) at common/admin_socket.cc:239
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67ebb40700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 116 (Thread 0x7f67eb33f700 (LWP 31552)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007f67efae6f33 in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67eb33f700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 115 (Thread 0x7f67e61e0700 (LWP 31563)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x31fd518) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x31fd000) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e61e0700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 114 (Thread 0x7f67e69e1700 (LWP 31564)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x31fdb18) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x31fd600) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e69e1700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 113 (Thread 0x7f67e71e2700 (LWP 31565)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x31fe718) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x31fe200) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e71e2700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 112 (Thread 0x7f67e91e6700 (LWP 31566)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x31fed18) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x31fe800) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e91e6700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 111 (Thread 0x7f67eab3e700 (LWP 31567)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x31fe118) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x31fdc00) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67eab3e700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 110 (Thread 0x7f67ea33d700 (LWP 31568)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a52f69 in Wait (mutex=..., this=0x3201718) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x3201200) at msg/SimpleMessenger.cc:210
#3  0x0000000000a58fbd in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67ea33d700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 109 (Thread 0x7f67e99e7700 (LWP 31569)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x36bf080) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x36bf070) at common/Timer.cc:114
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e99e7700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 108 (Thread 0x7f67e89e5700 (LWP 31570)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a62187 in Wait (mutex=..., this=0x36c1520) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x36c1510) at common/Timer.cc:112
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e89e5700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 107 (Thread 0x7f67e81e4700 (LWP 31571)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009053fc in Wait (mutex=..., this=0x31eeaa0) at ./common/Cond.h:55
#2  WBThrottle::get_next_should_flush (this=0x31ee998, next=0x7f67e81e3d00) at os/WBThrottle.cc:142
#3  0x0000000000905ca4 in WBThrottle::entry (this=0x31ee998) at os/WBThrottle.cc:160
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e81e4700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 106 (Thread 0x7f67e79e3700 (LWP 31572)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x000000000088b638 in WaitUntil (when=..., mutex=..., this=0x31ee680) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31ee680) at ./common/Cond.h:79
#3  FileStore::sync_entry (this=0x31ee000) at os/FileStore.cc:3097
#4  0x00000000008a98dd in FileStore::SyncThread::entry (this=<optimized out>) at os/FileStore.h:178
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67e79e3700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 105 (Thread 0x7f67e59df700 (LWP 31573)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000971e0d in Wait (mutex=..., this=0x31e8138) at ./common/Cond.h:55
#2  FileJournal::write_thread_entry (this=0x31e8000) at os/FileJournal.cc:1109
#3  0x00000000008a324d in FileJournal::Writer::entry (this=<optimized out>) at os/FileJournal.h:334
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e59df700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 104 (Thread 0x7f67e51de700 (LWP 31574)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009697d9 in Wait (mutex=..., this=0x31e8380) at ./common/Cond.h:55
#2  FileJournal::write_finish_thread_entry (this=0x31e8000) at os/FileJournal.cc:1333
#3  0x00000000008a322d in FileJournal::WriteFinisher::entry (this=<optimized out>) at os/FileJournal.h:344
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67e51de700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 103 (Thread 0x7f67e49dd700 (LWP 31575)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x31ee080) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x31ee020) at common/Finisher.cc:78
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67e49dd700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 102 (Thread 0x7f67e41dc700 (LWP 31576)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x31eee08) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31eee08) at common/Cond.h:79
#3  ThreadPool::worker (this=0x31eed90, wt=0x3560080) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67e41dc700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 101 (Thread 0x7f67e39db700 (LWP 31577)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x31eee08) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x31eee08) at common/Cond.h:79
#3  ThreadPool::worker (this=0x31eed90, wt=0x35602c0) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67e39db700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 100 (Thread 0x7f67e31da700 (LWP 31578)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x31eecc0) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x31eec60) at common/Finisher.cc:78
---Type <return> to continue, or q <return> to quit---
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67e31da700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 99 (Thread 0x7f67e29d9700 (LWP 31579)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x31ee550) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x31ee4f0) at common/Finisher.cc:78
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67e29d9700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 98 (Thread 0x7f67dfb9e700 (LWP 31581)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fd150) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fd0e8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67dfb9e700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 97 (Thread 0x7f67df39d700 (LWP 31582)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b0b3b4 in Accepter::entry (this=0x31fd0c0) at msg/Accepter.cc:196
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67df39d700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 96 (Thread 0x7f67deb9c700 (LWP 31583)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fdd50) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fdce8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67deb9c700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 95 (Thread 0x7f67ddb9a700 (LWP 31585)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fd750) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fd6e8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67ddb9a700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 94 (Thread 0x7f67dd399700 (LWP 31586)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fe350) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fe2e8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67dd399700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 93 (Thread 0x7f67dc397700 (LWP 31588)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x31fe950) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x31fe8e8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67dc397700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 92 (Thread 0x7f67db395700 (LWP 31590)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b1d3c8 in Wait (mutex=..., this=0x3201350) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x32012e8) at msg/DispatchQueue.cc:138
#3  0x0000000000a585cd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67db395700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 91 (Thread 0x7f67dab94700 (LWP 31591)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b0b3b4 in Accepter::entry (this=0x32012c0) at msg/Accepter.cc:196
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67dab94700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 90 (Thread 0x7f67da393700 (LWP 31592)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x7fff640b45d8) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7fff640b45c8) at common/Timer.cc:114
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67da393700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 89 (Thread 0x7f67d9b92700 (LWP 31593)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x7fff640b46f0) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7fff640b4690) at common/Finisher.cc:78
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d9b92700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 88 (Thread 0x7f67d9391700 (LWP 31594)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf4e8) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf4e8) at common/Cond.h:79
#3  ThreadPool::worker (this=0x36bf470, wt=0x43d3720) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d9391700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 87 (Thread 0x7f67d8b90700 (LWP 31595)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf4e8) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf4e8) at common/Cond.h:79
#3  ThreadPool::worker (this=0x36bf470, wt=0x4391d80) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d8b90700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 86 (Thread 0x7f67d838f700 (LWP 31596)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf670) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf670) at common/Cond.h:79
#3  ThreadPool::worker (this=0x36bf5f8, wt=0x43d3600) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d838f700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 85 (Thread 0x7f67d7b8e700 (LWP 31597)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf7f8) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf7f8) at common/Cond.h:79
#3  ThreadPool::worker (this=0x36bf780, wt=0x43d36a0) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d7b8e700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 84 (Thread 0x7f67d738d700 (LWP 31598)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a6581e in WaitUntil (when=..., mutex=..., this=0x36bf980) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bf980) at common/Cond.h:79
#3  ThreadPool::worker (this=0x36bf908, wt=0x43d3400) at common/WorkQueue.cc:143
#4  0x0000000000a671a0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d738d700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 83 (Thread 0x7f67d6b8c700 (LWP 31599)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x000000000063507b in WaitUntil (when=..., mutex=..., this=0x36bfb20) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x36bfb20) at ./common/Cond.h:79
#3  OSD::heartbeat_entry (this=0x36bf000) at osd/OSD.cc:3084
#4  0x000000000068e87d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:1041
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d6b8c700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 82 (Thread 0x7f67d638b700 (LWP 31600)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x36c1698) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x36c1638) at common/Finisher.cc:78
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d638b700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 81 (Thread 0x7f67d5b8a700 (LWP 31601)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009b5c80 in Wait (mutex=..., this=0x36c12a8) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x36c1248) at common/Finisher.cc:78
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d5b8a700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---

Thread 80 (Thread 0x7f67d5389700 (LWP 31602)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000a626db in WaitUntil (when=..., mutex=..., this=0x36c0f88) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x36c0f78) at common/Timer.cc:114
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d5389700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 79 (Thread 0x7f67d4b88700 (LWP 31604)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a62187 in Wait (mutex=..., this=0x36c13f8) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x36c13e8) at common/Timer.cc:112
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d4b88700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 78 (Thread 0x7f67d4387700 (LWP 31605)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000a62187 in Wait (mutex=..., this=0x36c0e68) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x36c0e58) at common/Timer.cc:112
#3  0x0000000000a6308d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d4387700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 77 (Thread 0x7f67d3b86700 (LWP 31606)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000615664 in Wait (mutex=..., this=<optimized out>) at ./common/Cond.h:55
#2  OSDService::agent_entry (this=0x36c06b8) at osd/OSD.cc:514
#3  0x0000000000673a2d in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:465
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d3b86700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 76 (Thread 0x7f67d0b79700 (LWP 31623)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00000000009951e4 in SignalHandler::entry (this=0x3dc4780) at global/signal_handler.cc:213
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67d0b79700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 75 (Thread 0x7f67bf96d700 (LWP 12970)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b0b3b4 in Accepter::entry (this=0x31fdcc0) at msg/Accepter.cc:196
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67bf96d700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 74 (Thread 0x7f67dbb96700 (LWP 12971)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b0b3b4 in Accepter::entry (this=0x31fe8c0) at msg/Accepter.cc:196
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67dbb96700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 73 (Thread 0x7f67dcb98700 (LWP 12972)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b0b3b4 in Accepter::entry (this=0x31fe2c0) at msg/Accepter.cc:196
#2  0x00007f67ef62be9a in start_thread (arg=0x7f67dcb98700) at pthread_create.c:308
#3  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 72 (Thread 0x7f67c3c3d700 (LWP 12982)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x48f5970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x48f5780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c3c3d700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 71 (Thread 0x7f67ce568700 (LWP 12983)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3b21e70) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3b21c80) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67ce568700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 70 (Thread 0x7f67be062700 (LWP 12984)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x464c1f0) at ./common/Cond.h:55
---Type <return> to continue, or q <return> to quit---
#2  Pipe::writer (this=0x464c000) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67be062700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 69 (Thread 0x7f67cfc71700 (LWP 12985)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x464cbf0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x464ca00) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67cfc71700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 68 (Thread 0x7f67c854e700 (LWP 12987)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3926470) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3926280) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c854e700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 67 (Thread 0x7f67be567700 (LWP 12988)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3926970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3926780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67be567700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 66 (Thread 0x7f67de29a700 (LWP 12992)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x526abf0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x526aa00) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67de29a700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 65 (Thread 0x7f67de199700 (LWP 12995)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x526a470) at ./common/Cond.h:55
#2  Pipe::writer (this=0x526a280) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67de199700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 64 (Thread 0x7f67ddf97700 (LWP 13005)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x48f5780, buf=0x7f67ddf96e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x48f5780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67ddf97700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 63 (Thread 0x7f67ca855700 (LWP 13006)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x3f54540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3f54480) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67ca855700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 62 (Thread 0x7f67dde96700 (LWP 13007)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21c80, buf=0x7f67dde95e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3b21c80) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67dde96700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 61 (Thread 0x7f67cd263700 (LWP 13008)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x3f540c0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3f54000) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67cd263700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---

Thread 60 (Thread 0x7f67ddd95700 (LWP 13009)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x464ca00, buf=0x7f67ddd94e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x464ca00) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67ddd95700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 59 (Thread 0x7f67d3284700 (LWP 13010)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3926280, buf=0x7f67d3283e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3926280) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d3284700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 58 (Thread 0x7f67ce467700 (LWP 13011)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x3f54c00) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3f54b40) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67ce467700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 57 (Thread 0x7f67cf76c700 (LWP 13012)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x3f54420) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3f54360) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67cf76c700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 56 (Thread 0x7f67d2781700 (LWP 13013)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x464c000, buf=0x7f67d2780e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x464c000) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d2781700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 55 (Thread 0x7f67d237c700 (LWP 13014)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3926780, buf=0x7f67d237be1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3926780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d237c700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 54 (Thread 0x7f67d1d7d700 (LWP 13015)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x444a540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x444a480) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d1d7d700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 53 (Thread 0x7f67d137a700 (LWP 13016)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x444a300) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x444a240) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d137a700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 52 (Thread 0x7f67d0378700 (LWP 13020)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x526aa00, buf=0x7f67d0377e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x526aa00) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d0378700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 51 (Thread 0x7f67d0176700 (LWP 13019)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
---Type <return> to continue, or q <return> to quit---
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x526a280, buf=0x7f67d0175e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x526a280) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67d0176700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 50 (Thread 0x7f67cb857700 (LWP 13022)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d71e0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d7120) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67cb857700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 49 (Thread 0x7f67cdb65700 (LWP 13023)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d70c0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d7000) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67cdb65700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 48 (Thread 0x7f67cf86d700 (LWP 13027)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3d291f0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3d29000) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67cf86d700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 47 (Thread 0x7f67ce669700 (LWP 13028)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3d29000, buf=0x7f67ce668e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3d29000) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67ce669700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 46 (Thread 0x7f67c9853700 (LWP 13029)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d7ae0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d7a20) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c9853700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 45 (Thread 0x7f67c3839700 (LWP 13042)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x526ac80, buf=0x7f67c3838e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x526ac80) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c3839700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 44 (Thread 0x7f67bdd5f700 (LWP 13043)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x526a780, buf=0x7f67bdd5ee1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x526a780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67bdd5f700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 43 (Thread 0x7f67c2071700 (LWP 13044)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x5241000, buf=0x7f67c2070e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x5241000) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c2071700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 42 (Thread 0x7f67c52d2700 (LWP 13045)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x4585780, buf=0x7f67c52d1e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x4585780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
---Type <return> to continue, or q <return> to quit---
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c52d2700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 41 (Thread 0x7f67c3435700 (LWP 13046)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x5241a00, buf=0x7f67c3434e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x5241a00) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c3435700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 40 (Thread 0x7f67c50d0700 (LWP 13047)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x526ae70) at ./common/Cond.h:55
#2  Pipe::writer (this=0x526ac80) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c50d0700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 39 (Thread 0x7f67c734a700 (LWP 13048)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d7f60) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d7ea0) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c734a700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 38 (Thread 0x7f67c5945700 (LWP 13049)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x526a970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x526a780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c5945700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 37 (Thread 0x7f67c6b49700 (LWP 13050)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d7540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d7480) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c6b49700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 36 (Thread 0x7f67cbb5a700 (LWP 13051)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x5241bf0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x5241a00) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67cbb5a700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 35 (Thread 0x7f67c6348700 (LWP 13052)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d7660) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x35d75a0) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c6348700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 34 (Thread 0x7f67c4fcf700 (LWP 13053)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x4585970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x4585780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c4fcf700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 33 (Thread 0x7f67be365700 (LWP 13054)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x52411f0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x5241000) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67be365700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7f67c4ece700 (LWP 13055)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35d7780) at ./common/Cond.h:55
---Type <return> to continue, or q <return> to quit---
#2  Pipe::DelayedDelivery::entry (this=0x35d76c0) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c4ece700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 31 (Thread 0x7f67c46cd700 (LWP 13056)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x35610c0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3561000) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c46cd700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7f67c7f48700 (LWP 13057)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21780, buf=0x7f67c7f47e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3b21780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c7f48700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7f67c3637700 (LWP 13058)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3b21970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3b21780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67c3637700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7f67c2bb5700 (LWP 13059)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x3561300) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x3561240) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c2bb5700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f67d247d700 (LWP 13062)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3b216f0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3b21500) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d247d700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7f67c7b4b700 (LWP 13063)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x360d780) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x360d6c0) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c7b4b700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7f67c0d6c700 (LWP 13069)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x3b21280, buf=0x7f67c0d6be1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x3b21280) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c0d6c700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7f67bef6a700 (LWP 13070)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x3b21470) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3b21280) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67bef6a700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7f67d3183700 (LWP 13071)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x49a4420) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x49a4360) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67d3183700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7f67d217a700 (LWP 13085)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
---Type <return> to continue, or q <return> to quit---
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x42eb970) at ./common/Cond.h:55
#2  Pipe::writer (this=0x42eb780) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67d217a700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f67cca62700 (LWP 13087)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x42eb780, buf=0x7f67cca61e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x42eb780) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67cca62700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f67c1a6b700 (LWP 13090)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x360d540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x360d480) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67c1a6b700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f67bd059700 (LWP 13280)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x5241470) at ./common/Cond.h:55
#2  Pipe::writer (this=0x5241280) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67bd059700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f67bd15a700 (LWP 13284)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x5241280, buf=0x7f67bd159e1a "\377", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x5241280) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67bd15a700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f67c3738700 (LWP 13437)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x4f99500, buf=0x7f67c3737e1a "\377\357g\177", len=1) at msg/Pipe.cc:2255
#3  0x0000000000b44620 in Pipe::reader (this=0x4f99500) at msg/Pipe.cc:1421
#4  0x0000000000b4859d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49
#5  0x00007f67ef62be9a in start_thread (arg=0x7f67c3738700) at pthread_create.c:308
#6  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f67cfa6f700 (LWP 13438)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b3d103 in Wait (mutex=..., this=0x4f996f0) at ./common/Cond.h:55
#2  Pipe::writer (this=0x4f99500) at msg/Pipe.cc:1730
#3  0x0000000000b4857d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61
#4  0x00007f67ef62be9a in start_thread (arg=0x7f67cfa6f700) at pthread_create.c:308
#5  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f67ca054700 (LWP 13439)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000b2d099 in Wait (mutex=..., this=0x360d420) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x360d360) at msg/Pipe.cc:209
#3  0x00007f67ef62be9a in start_thread (arg=0x7f67ca054700) at pthread_create.c:308
#4  0x00007f67edfdc3fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f67c1b6c700 (LWP 13442)):
#0  0x00007f67edfd0a43 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000b30832 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2282
#2  0x0000000000b30b20 in Pipe::tcp_read (this=0x4585000, buf=0x7f67c1b6be1a "\377", len=1) at ms
Actions
Copy link
 #2

Updated by Sage Weil over 9 years ago

..and when i detached gdb the osd saw it was marked down, and came back to life after that. :/

Actions
Copy link
 #3

Updated by Sage Weil over 9 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Sage Weil over 9 years ago

  • Status changed from New to Need More Info
Actions
Copy link
 #5

Updated by Sage Weil over 9 years ago

  • Priority changed from Urgent to High
Actions
Copy link
 #6

Updated by Samuel Just about 9 years ago

  • Status changed from Need More Info to Can't reproduce
Actions
Copy link

Also available in: Atom PDF