Project

General

Profile

Actions
Copy link

Bug #12745

closed

ms/simple: Messenger/MessengerTest.SyntheticInjectTest2/1 hang

Added by Sage Weil over 8 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023384

ceph_test_msgr

2015-08-20T10:32:47.380 INFO:teuthology.orchestra.run.mira059.stdout:[ RUN      ] Messenger/MessengerTest.SyntheticInjectTest2/1
2015-08-20T10:32:47.380 INFO:teuthology.orchestra.run.mira059.stderr:SetUp start set up simple
2015-08-20T10:32:47.422 INFO:teuthology.orchestra.run.mira059.stderr:seeding connection 0
2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428614 7ffa4314e700  0 -- 127.0.0.1:16821/3366 >> :/0 pipe(0x7ffa64855000 sd=40 :0 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60ba20).injecting socket failure
2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428790 7ffa0cd5d700  0 -- 127.0.0.1:16802/3347 >> 127.0.0.1:16821/3366 pipe(0x7ffa9a309000 sd=39 :16802 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60d700).accept connect_seq 0 vs existing 0 state connecting
2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428822 7ffa0cc5c700  0 -- 127.0.0.1:16823/3368 >> 127.0.0.1:16802/3347 pipe(0x7ffa722e8000 sd=41 :16823 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60cc00).accept connect_seq 0 vs existing 0 state connecting
...
2015-08-20T10:33:38.522 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12
2015-08-20T10:33:43.531 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12
2015-08-20T10:33:48.541 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12
2015-08-20T10:33:53.549 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12
2015-08-20T10:33:58.559 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12
...
Actions
Copy link
 #1

Updated by Sage Weil over 8 years ago 


Thread 380 (Thread 0x7ffa3fe3d700 (LWP 3346)):
#0  0x00007ffa4244f8ad in recvmsg () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa42a870ed in ustcomm_recv_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#2  0x00007ffa42a89a3c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa3fe3d700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 379 (Thread 0x7ffa3f63c700 (LWP 3347)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffa42a8a3cb in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
#2  0x00007ffa42448182 in start_thread (arg=0x7ffa3f63c700) at pthread_create.c:312
#3  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 378 (Thread 0x7ffa3ee3b700 (LWP 3348)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa432f96eb in ceph::log::Log::entry (this=0x7ffa46d7e160) at log/Log.cc:353
#2  0x00007ffa42448182 in start_thread (arg=0x7ffa3ee3b700) at pthread_create.c:312
#3  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 377 (Thread 0x7ffa3dec0700 (LWP 3349)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ffa433b3a3a in WaitUntil (when=..., mutex=..., this=0x7ffa46d8efe8) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7ffa46d8efe8) at ./common/Cond.h:79
#3  CephContextServiceThread::entry (this=0x7ffa46d8ef70) at common/ceph_context.cc:96
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3dec0700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 376 (Thread 0x7ffa3d6bf700 (LWP 3350)):
#0  0x00007ffa40fd26a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa434a0df2 in EpollDriver::event_wait (this=0x7ffa46d79420, fired_events=..., tvp=<optimized out>) at msg/async/EventEpoll.cc:115
#2  0x00007ffa4349da5c in EventCenter::process_events (this=this@entry=0x7ffa46da2040, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:360
#3  0x00007ffa434692e8 in Worker::entry (this=0x7ffa46da2000) at msg/async/AsyncMessenger.cc:301
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3d6bf700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 375 (Thread 0x7ffa3cebe700 (LWP 3351)):
#0  0x00007ffa40fd26a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa434a0df2 in EpollDriver::event_wait (this=0x7ffa46d793c0, fired_events=..., tvp=<optimized out>) at msg/async/EventEpoll.cc:115
#2  0x00007ffa4349da5c in EventCenter::process_events (this=this@entry=0x7ffa46da2700, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:360
#3  0x00007ffa434692e8 in Worker::entry (this=0x7ffa46da26c0) at msg/async/AsyncMessenger.cc:301
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3cebe700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 374 (Thread 0x7ffa0ee62700 (LWP 16445)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac2ac30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa8ac2abc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0ee62700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 373 (Thread 0x7ffa0fe64700 (LWP 16446)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac2add0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa8ac2abc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0fe64700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 372 (Thread 0x7ffa0f663700 (LWP 16447)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa0f662c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa8ac2ab90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa0f663700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 371 (Thread 0x7ffa0be5c700 (LWP 16448)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2b038) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa8ac2aa00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0be5c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 370 (Thread 0x7ffa3c6bd700 (LWP 16449)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac2a530) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa8ac2a4c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3c6bd700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 369 (Thread 0x7ffa3bebc700 (LWP 16450)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac2a6d0) at ./common/Cond.h:55
---Type <return> to continue, or q <return> to quit---
#2  DispatchQueue::run_local_delivery (this=0x7ffa8ac2a4c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3bebc700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 368 (Thread 0x7ffa3b6bb700 (LWP 16451)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa3b6bac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa8ac2a490) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa3b6bb700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 367 (Thread 0x7ffa3aeba700 (LWP 16452)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2a938) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa8ac2a300) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3aeba700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 366 (Thread 0x7ffa3a6b9700 (LWP 16453)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac29e30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa8ac29dc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa3a6b9700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 365 (Thread 0x7ffa39eb8700 (LWP 16454)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac29fd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa8ac29dc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa39eb8700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 364 (Thread 0x7ffa396b7700 (LWP 16455)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa396b6c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa8ac29d90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa396b7700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 363 (Thread 0x7ffa38eb6700 (LWP 16456)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2a238) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa8ac29c00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa38eb6700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 362 (Thread 0x7ffa386b5700 (LWP 16457)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7d330) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7d2c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa386b5700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 361 (Thread 0x7ffa37eb4700 (LWP 16458)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7d4d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7d2c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa37eb4700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 360 (Thread 0x7ffa376b3700 (LWP 16459)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa376b2c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7d290) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa376b3700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 359 (Thread 0x7ffa36eb2700 (LWP 16460)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7d738) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7d100) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa36eb2700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 358 (Thread 0x7ffa366b1700 (LWP 16461)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7c530) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7c4c8) at msg/simple/DispatchQueue.cc:196
---Type <return> to continue, or q <return> to quit---
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa366b1700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 357 (Thread 0x7ffa35eb0700 (LWP 16462)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7c6d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7c4c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa35eb0700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 356 (Thread 0x7ffa356af700 (LWP 16463)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa356aec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7c490) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa356af700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 355 (Thread 0x7ffa34eae700 (LWP 16464)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7c938) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7c300) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa34eae700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 354 (Thread 0x7ffa346ad700 (LWP 16465)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7be30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7bdc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa346ad700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 353 (Thread 0x7ffa33eac700 (LWP 16466)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7bfd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7bdc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa33eac700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 352 (Thread 0x7ffa336ab700 (LWP 16467)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa336aac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7bd90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa336ab700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 351 (Thread 0x7ffa32eaa700 (LWP 16468)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7c238) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7bc00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa32eaa700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 350 (Thread 0x7ffa326a9700 (LWP 16469)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7b730) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7b6c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa326a9700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 349 (Thread 0x7ffa31ea8700 (LWP 16470)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7b8d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7b6c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa31ea8700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 348 (Thread 0x7ffa316a7700 (LWP 16471)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa316a6c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7b690) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa316a7700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 347 (Thread 0x7ffa30ea6700 (LWP 16472)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7bb38) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7b500) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
---Type <return> to continue, or q <return> to quit---
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa30ea6700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 346 (Thread 0x7ffa306a5700 (LWP 16473)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7b030) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7afc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa306a5700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 345 (Thread 0x7ffa2fea4700 (LWP 16474)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7b1d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7afc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2fea4700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 344 (Thread 0x7ffa2f6a3700 (LWP 16475)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2f6a2c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7af90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa2f6a3700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 343 (Thread 0x7ffa2eea2700 (LWP 16476)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7b438) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7ae00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2eea2700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 342 (Thread 0x7ffa2e6a1700 (LWP 16477)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7a930) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7a8c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2e6a1700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 341 (Thread 0x7ffa2dea0700 (LWP 16478)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7aad0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7a8c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2dea0700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 340 (Thread 0x7ffa2d69f700 (LWP 16479)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2d69ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7a890) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa2d69f700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 339 (Thread 0x7ffa2ce9e700 (LWP 16480)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7ad38) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7a700) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2ce9e700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 338 (Thread 0x7ffa2c69d700 (LWP 16481)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7cc30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7cbc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2c69d700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 337 (Thread 0x7ffa2be9c700 (LWP 16482)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7cdd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7cbc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2be9c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 336 (Thread 0x7ffa2b69b700 (LWP 16483)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2b69ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7cb90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa2b69b700) at pthread_create.c:312
---Type <return> to continue, or q <return> to quit---
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 335 (Thread 0x7ffa2ae9a700 (LWP 16484)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7d038) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7ca00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2ae9a700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 334 (Thread 0x7ffa2a699700 (LWP 16485)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7da30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa46d7d9c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2a699700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 333 (Thread 0x7ffa29e98700 (LWP 16486)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7dbd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa46d7d9c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa29e98700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 332 (Thread 0x7ffa29697700 (LWP 16487)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa29696c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa46d7d990) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa29697700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 331 (Thread 0x7ffa28e96700 (LWP 16488)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7de38) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa46d7d800) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa28e96700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 330 (Thread 0x7ffa28695700 (LWP 16489)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6de530) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa6f6de4c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa28695700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 329 (Thread 0x7ffa27e94700 (LWP 16490)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6de6d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa6f6de4c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa27e94700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 328 (Thread 0x7ffa27693700 (LWP 16491)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa27692c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa6f6de490) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa27693700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 327 (Thread 0x7ffa26e92700 (LWP 16492)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6de938) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa6f6de300) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa26e92700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 326 (Thread 0x7ffa26691700 (LWP 16493)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dc230) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa6f6dc1c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa26691700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 325 (Thread 0x7ffa25e90700 (LWP 16494)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6dc3d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa6f6dc1c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa25e90700) at pthread_create.c:312
---Type <return> to continue, or q <return> to quit---
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 324 (Thread 0x7ffa2568f700 (LWP 16495)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2568ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa6f6dc190) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa2568f700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 323 (Thread 0x7ffa24e8e700 (LWP 16496)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6dc638) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa6f6dc000) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa24e8e700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 322 (Thread 0x7ffa2468d700 (LWP 16497)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dd030) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa6f6dcfc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa2468d700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 321 (Thread 0x7ffa23e8c700 (LWP 16498)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6dd1d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa6f6dcfc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa23e8c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 320 (Thread 0x7ffa2368b700 (LWP 16499)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2368ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa6f6dcf90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa2368b700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 319 (Thread 0x7ffa22e8a700 (LWP 16500)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6dd438) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa6f6dce00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa22e8a700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 318 (Thread 0x7ffa22689700 (LWP 16501)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dde30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffa6f6dddc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa22689700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 317 (Thread 0x7ffa21e88700 (LWP 16502)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6ddfd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffa6f6dddc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa21e88700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 316 (Thread 0x7ffa21687700 (LWP 16503)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa21686c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffa6f6ddd90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa21687700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 315 (Thread 0x7ffa20e86700 (LWP 16504)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6de238) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffa6f6ddc00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa20e86700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 314 (Thread 0x7ffa20685700 (LWP 16505)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4962230) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffab49621c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa20685700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
---Type <return> to continue, or q <return> to quit---

Thread 313 (Thread 0x7ffa1fe84700 (LWP 16506)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49623d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffab49621c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1fe84700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 312 (Thread 0x7ffa1f683700 (LWP 16507)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1f682c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffab4962190) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa1f683700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 311 (Thread 0x7ffa1ee82700 (LWP 16508)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4962638) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffab4962000) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1ee82700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 310 (Thread 0x7ffa1e681700 (LWP 16509)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4964530) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffab49644c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1e681700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 309 (Thread 0x7ffa1de80700 (LWP 16510)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49646d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffab49644c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1de80700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 308 (Thread 0x7ffa1d67f700 (LWP 16511)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1d67ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffab4964490) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa1d67f700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 307 (Thread 0x7ffa1ce7e700 (LWP 16512)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4964938) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffab4964300) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1ce7e700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 306 (Thread 0x7ffa1c67d700 (LWP 16513)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4963030) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffab4962fc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1c67d700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 305 (Thread 0x7ffa1be7c700 (LWP 16514)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49631d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffab4962fc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1be7c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 304 (Thread 0x7ffa1b67b700 (LWP 16515)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1b67ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffab4962f90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa1b67b700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 303 (Thread 0x7ffa1ae7a700 (LWP 16516)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4963438) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffab4962e00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1ae7a700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

---Type <return> to continue, or q <return> to quit---
Thread 302 (Thread 0x7ffa1a679700 (LWP 16517)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21730) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e216c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1a679700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 301 (Thread 0x7ffa19e78700 (LWP 16518)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e218d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e216c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa19e78700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 300 (Thread 0x7ffa19677700 (LWP 16519)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa19676c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e21690) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa19677700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 299 (Thread 0x7ffa18e76700 (LWP 16520)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e21b38) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e21500) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa18e76700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 298 (Thread 0x7ffa18675700 (LWP 16521)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21e30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e21dc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa18675700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 297 (Thread 0x7ffa17e74700 (LWP 16522)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e21fd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e21dc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa17e74700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 296 (Thread 0x7ffa17673700 (LWP 16523)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa17672c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e21d90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa17673700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 295 (Thread 0x7ffa16e72700 (LWP 16524)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e22238) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e21c00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa16e72700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 294 (Thread 0x7ffa16671700 (LWP 16525)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e23a30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e239c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa16671700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 293 (Thread 0x7ffa15e70700 (LWP 16526)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e23bd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e239c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa15e70700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 292 (Thread 0x7ffa1566f700 (LWP 16527)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1566ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e23990) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa1566f700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 291 (Thread 0x7ffa14e6e700 (LWP 16528)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23e38) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e23800) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa14e6e700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 290 (Thread 0x7ffa1466d700 (LWP 16529)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e23330) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e232c8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa1466d700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 289 (Thread 0x7ffa13e6c700 (LWP 16530)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e234d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e232c8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa13e6c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 288 (Thread 0x7ffa1366b700 (LWP 16531)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1366ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e23290) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa1366b700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 287 (Thread 0x7ffa12e6a700 (LWP 16532)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23738) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e23100) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa12e6a700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 286 (Thread 0x7ffa12669700 (LWP 16533)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e22c30) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e22bc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa12669700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 285 (Thread 0x7ffa11e68700 (LWP 16534)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e22dd0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e22bc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa11e68700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 284 (Thread 0x7ffa11667700 (LWP 16535)):
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa11666c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e22b90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa11667700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 283 (Thread 0x7ffa10e66700 (LWP 16536)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23038) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e22a00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa10e66700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 282 (Thread 0x7ffa10665700 (LWP 16537)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21030) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x7ffae2e20fc8) at msg/simple/DispatchQueue.cc:196
#3  0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa10665700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 281 (Thread 0x7ffa0e661700 (LWP 16538)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e211d0) at ./common/Cond.h:55
#2  DispatchQueue::run_local_delivery (this=0x7ffae2e20fc8) at msg/simple/DispatchQueue.cc:114
#3  0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0e661700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 280 (Thread 0x7ffa0de60700 (LWP 16539)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa0de5fc60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
#2  Accepter::entry (this=0x7ffae2e20f90) at msg/simple/Accepter.cc:216
#3  0x00007ffa42448182 in start_thread (arg=0x7ffa0de60700) at pthread_create.c:312
#4  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 279 (Thread 0x7ffa0d65f700 (LWP 16540)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e21438) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x7ffae2e20e00) at msg/simple/SimpleMessenger.cc:212
#3  0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0d65f700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 278 (Thread 0x7ffa0c252700 (LWP 16561)):

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434d62aa in Wait (mutex=..., this=0x7ffab4949228) at ./common/Cond.h:55
#2  Pipe::reader (this=0x7ffab4945000) at msg/simple/Pipe.cc:1512
#3  0x00007ffa434df75d in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0c252700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 277 (Thread 0x7ffa0b459700 (LWP 16569)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffab4949228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffab4945000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0b459700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 276 (Thread 0x7ffa0a74c700 (LWP 16583)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434d62aa in Wait (mutex=..., this=0x7ffa6fabb228) at ./common/Cond.h:55
#2  Pipe::reader (this=0x7ffa6fab7000) at msg/simple/Pipe.cc:1512
#3  0x00007ffa434df75d in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa0a74c700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 275 (Thread 0x7ffa09a3f700 (LWP 16609)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa55a78228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffa55a74000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa09a3f700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 274 (Thread 0x7ffa09136700 (LWP 16621)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa6946d228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffa69469000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa09136700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 273 (Thread 0x7ffa09035700 (LWP 16622)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa551f0228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffa551ec000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa09035700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 272 (Thread 0x7ffa09439700 (LWP 16643)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa69383228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffa6937f000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa09439700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 271 (Thread 0x7ffa07e23700 (LWP 16647)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa6fabb228) at ./common/Cond.h:55
#2  Pipe::writer (this=0x7ffa6fab7000) at msg/simple/Pipe.cc:1845
#3  0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62
#4  0x00007ffa42448182 in start_thread (arg=0x7ffa07e23700) at pthread_create.c:312
#5  0x00007ffa40fd200d in clone () at ../sysdeps/unix/sys
Actions
Copy link
 #2

Updated by Haomai Wang over 8 years ago

Hmm, it seemed like a rare case, I can't reproduce this in local env.

Actions
Copy link
 #3

Updated by Sage Weil over 8 years ago

I think the test is buggy. It randomly does drop_connection(), which will mark_down() one end of a stateful lossless_peer connection. In reality the other end will also mark_down...

The actual race is this:

- A queues messages to B
 - fault
 - A and B race connecting to each other
 - A sends WAIT and waits for B to connect to it
 - B does mark_down

which means B never connects but A is waiting forever for it to.

We could work around this by making the connection race WAIT state retry periodically, but in reality this shouldn't happen, I think!

Actions
Copy link
 #4

Updated by Sage Weil over 8 years ago 

2015-08-25 14:43:32.950307 7f33c3e24700  2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault (32) Broken pipe
2015-08-25 14:43:32.950319 7f33c3e24700  0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault
2015-08-25 14:43:32.950326 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer: state = connecting policy.server=0
2015-08-25 14:43:32.950334 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect 7
2015-08-25 14:43:32.950367 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connecting to 127.0.0.1:16820/29166
2015-08-25 14:43:32.950721 7f33c3e24700  2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect couldn't read banner, (32) Broken pipe
2015-08-25 14:43:32.950763 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580). sleep for 0.1
2015-08-25 14:43:32.951018 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept peer addr is 127.0.0.1:16820/29166
2015-08-25 14:43:32.951035 7f33c341a700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept got peer connect_seq 7 global_seq 20
2015-08-25 14:43:32.951044 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept of host_type 8, policy.lossy=0 policy.server=0 policy.standby=1 policy.resetcheck=1
2015-08-25 14:43:32.951049 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept my proto 24, their proto 24
2015-08-25 14:43:32.951053 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept:  setting up session_security.
2015-08-25 14:43:32.951059 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept existing 0x7f34148fb000.gseq 17 <= 20, looks ok
2015-08-25 14:43:32.951064 7f33c341a700  0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connect_seq 7 vs existing 7 state connecting
2015-08-25 14:43:32.951070 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connection race, existing 0x7f34148fb000.cseq 7 == 7, sending WAIT

2015-08-25 14:43:32.967993 7f3403770780  1 -- 127.0.0.1:16801/29147 mark_down 0x7f3414a1c580 -- 0x7f34148fb000
2015-08-25 14:43:32.968001 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).unregister_pipe
2015-08-25 14:43:32.968010 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).stop

2015-08-25 14:43:33.050859 7f33c3e24700  3 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect fault, but state = closed != connecting, stopping
2015-08-25 14:43:33.050893 7f33c3e24700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer finishing
2015-08-25 14:43:33.050918 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer done
Actions
Copy link
 #5

Updated by Sage Weil over 8 years ago

  • Status changed from New to 4
2015-08-25 14:43:32.950307 7f33c3e24700  2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault (32) Broken pipe
2015-08-25 14:43:32.950319 7f33c3e24700  0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault
2015-08-25 14:43:32.950326 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer: state = connecting policy.server=0
2015-08-25 14:43:32.950334 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect 7
2015-08-25 14:43:32.950367 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connecting to 127.0.0.1:16820/29166
2015-08-25 14:43:32.950721 7f33c3e24700  2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect couldn't read banner, (32) Broken pipe
2015-08-25 14:43:32.950763 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580). sleep for 0.1
2015-08-25 14:43:32.951018 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept peer addr is 127.0.0.1:16820/29166
2015-08-25 14:43:32.951035 7f33c341a700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept got peer connect_seq 7 global_seq 20
2015-08-25 14:43:32.951044 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept of host_type 8, policy.lossy=0 policy.server=0 policy.standby=1 policy.resetcheck=1
2015-08-25 14:43:32.951049 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept my proto 24, their proto 24
2015-08-25 14:43:32.951053 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept:  setting up session_security.
2015-08-25 14:43:32.951059 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept existing 0x7f34148fb000.gseq 17 <= 20, looks ok
2015-08-25 14:43:32.951064 7f33c341a700  0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connect_seq 7 vs existing 7 state connecting
2015-08-25 14:43:32.951070 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connection race, existing 0x7f34148fb000.cseq 7 == 7, sending WAIT

2015-08-25 14:43:32.967993 7f3403770780  1 -- 127.0.0.1:16801/29147 mark_down 0x7f3414a1c580 -- 0x7f34148fb000
2015-08-25 14:43:32.968001 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).unregister_pipe
2015-08-25 14:43:32.968010 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).stop

2015-08-25 14:43:33.050859 7f33c3e24700  3 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect fault, but state = closed != connecting, stopping
2015-08-25 14:43:33.050893 7f33c3e24700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer finishing
2015-08-25 14:43:33.050918 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer done
Actions
Copy link
 #6

Updated by Sage Weil over 8 years ago

  • Status changed from 4 to Resolved
Actions
Copy link
 #7

Updated by Greg Farnum about 5 years ago

  • Project changed from Ceph to Messengers
  • Category deleted (msgr)
Actions
Copy link

Also available in: Atom PDF