Actions
Bug #12745
closedms/simple: Messenger/MessengerTest.SyntheticInjectTest2/1 hang
Added by Sage Weil over 8 years ago. Updated about 5 years ago.
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/sage-2015-08-19_19:18:53-rados-wip-newstore-misc-distro-basic-multi/1023384
ceph_test_msgr
2015-08-20T10:32:47.380 INFO:teuthology.orchestra.run.mira059.stdout:[ RUN ] Messenger/MessengerTest.SyntheticInjectTest2/1 2015-08-20T10:32:47.380 INFO:teuthology.orchestra.run.mira059.stderr:SetUp start set up simple 2015-08-20T10:32:47.422 INFO:teuthology.orchestra.run.mira059.stderr:seeding connection 0 2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428614 7ffa4314e700 0 -- 127.0.0.1:16821/3366 >> :/0 pipe(0x7ffa64855000 sd=40 :0 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60ba20).injecting socket failure 2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428790 7ffa0cd5d700 0 -- 127.0.0.1:16802/3347 >> 127.0.0.1:16821/3366 pipe(0x7ffa9a309000 sd=39 :16802 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60d700).accept connect_seq 0 vs existing 0 state connecting 2015-08-20T10:32:47.423 INFO:teuthology.orchestra.run.mira059.stderr:2015-08-20 10:32:47.428822 7ffa0cc5c700 0 -- 127.0.0.1:16823/3368 >> 127.0.0.1:16802/3347 pipe(0x7ffa722e8000 sd=41 :16823 s=0 pgs=0 cs=0 l=0 c=0x7ffabf60cc00).accept connect_seq 0 vs existing 0 state connecting ... 2015-08-20T10:33:38.522 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12 2015-08-20T10:33:43.531 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12 2015-08-20T10:33:48.541 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12 2015-08-20T10:33:53.549 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12 2015-08-20T10:33:58.559 INFO:teuthology.orchestra.run.mira059.stderr:available_connections: 40 inflight messages: 12 ...
Updated by Sage Weil over 8 years ago
Thread 380 (Thread 0x7ffa3fe3d700 (LWP 3346)): #0 0x00007ffa4244f8ad in recvmsg () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa42a870ed in ustcomm_recv_unix_sock () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #2 0x00007ffa42a89a3c in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa3fe3d700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 379 (Thread 0x7ffa3f63c700 (LWP 3347)): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007ffa42a8a3cb in ?? () from /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 #2 0x00007ffa42448182 in start_thread (arg=0x7ffa3f63c700) at pthread_create.c:312 #3 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 378 (Thread 0x7ffa3ee3b700 (LWP 3348)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa432f96eb in ceph::log::Log::entry (this=0x7ffa46d7e160) at log/Log.cc:353 #2 0x00007ffa42448182 in start_thread (arg=0x7ffa3ee3b700) at pthread_create.c:312 #3 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 377 (Thread 0x7ffa3dec0700 (LWP 3349)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007ffa433b3a3a in WaitUntil (when=..., mutex=..., this=0x7ffa46d8efe8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7ffa46d8efe8) at ./common/Cond.h:79 #3 CephContextServiceThread::entry (this=0x7ffa46d8ef70) at common/ceph_context.cc:96 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3dec0700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 376 (Thread 0x7ffa3d6bf700 (LWP 3350)): #0 0x00007ffa40fd26a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa434a0df2 in EpollDriver::event_wait (this=0x7ffa46d79420, fired_events=..., tvp=<optimized out>) at msg/async/EventEpoll.cc:115 #2 0x00007ffa4349da5c in EventCenter::process_events (this=this@entry=0x7ffa46da2040, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:360 #3 0x00007ffa434692e8 in Worker::entry (this=0x7ffa46da2000) at msg/async/AsyncMessenger.cc:301 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3d6bf700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 375 (Thread 0x7ffa3cebe700 (LWP 3351)): #0 0x00007ffa40fd26a3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa434a0df2 in EpollDriver::event_wait (this=0x7ffa46d793c0, fired_events=..., tvp=<optimized out>) at msg/async/EventEpoll.cc:115 #2 0x00007ffa4349da5c in EventCenter::process_events (this=this@entry=0x7ffa46da2700, timeout_microseconds=timeout_microseconds@entry=30000000) at msg/async/Event.cc:360 #3 0x00007ffa434692e8 in Worker::entry (this=0x7ffa46da26c0) at msg/async/AsyncMessenger.cc:301 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3cebe700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 374 (Thread 0x7ffa0ee62700 (LWP 16445)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac2ac30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa8ac2abc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0ee62700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 373 (Thread 0x7ffa0fe64700 (LWP 16446)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac2add0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa8ac2abc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0fe64700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 372 (Thread 0x7ffa0f663700 (LWP 16447)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa0f662c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa8ac2ab90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa0f663700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 371 (Thread 0x7ffa0be5c700 (LWP 16448)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2b038) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa8ac2aa00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0be5c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 370 (Thread 0x7ffa3c6bd700 (LWP 16449)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac2a530) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa8ac2a4c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3c6bd700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 369 (Thread 0x7ffa3bebc700 (LWP 16450)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac2a6d0) at ./common/Cond.h:55 ---Type <return> to continue, or q <return> to quit--- #2 DispatchQueue::run_local_delivery (this=0x7ffa8ac2a4c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3bebc700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 368 (Thread 0x7ffa3b6bb700 (LWP 16451)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa3b6bac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa8ac2a490) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa3b6bb700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 367 (Thread 0x7ffa3aeba700 (LWP 16452)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2a938) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa8ac2a300) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3aeba700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 366 (Thread 0x7ffa3a6b9700 (LWP 16453)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa8ac29e30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa8ac29dc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa3a6b9700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 365 (Thread 0x7ffa39eb8700 (LWP 16454)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa8ac29fd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa8ac29dc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa39eb8700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 364 (Thread 0x7ffa396b7700 (LWP 16455)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa396b6c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa8ac29d90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa396b7700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 363 (Thread 0x7ffa38eb6700 (LWP 16456)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa8ac2a238) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa8ac29c00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa38eb6700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 362 (Thread 0x7ffa386b5700 (LWP 16457)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7d330) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7d2c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa386b5700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 361 (Thread 0x7ffa37eb4700 (LWP 16458)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7d4d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7d2c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa37eb4700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 360 (Thread 0x7ffa376b3700 (LWP 16459)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa376b2c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7d290) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa376b3700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 359 (Thread 0x7ffa36eb2700 (LWP 16460)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7d738) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7d100) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa36eb2700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 358 (Thread 0x7ffa366b1700 (LWP 16461)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7c530) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7c4c8) at msg/simple/DispatchQueue.cc:196 ---Type <return> to continue, or q <return> to quit--- #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa366b1700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 357 (Thread 0x7ffa35eb0700 (LWP 16462)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7c6d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7c4c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa35eb0700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 356 (Thread 0x7ffa356af700 (LWP 16463)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa356aec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7c490) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa356af700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 355 (Thread 0x7ffa34eae700 (LWP 16464)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7c938) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7c300) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa34eae700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 354 (Thread 0x7ffa346ad700 (LWP 16465)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7be30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7bdc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa346ad700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 353 (Thread 0x7ffa33eac700 (LWP 16466)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7bfd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7bdc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa33eac700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 352 (Thread 0x7ffa336ab700 (LWP 16467)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa336aac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7bd90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa336ab700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 351 (Thread 0x7ffa32eaa700 (LWP 16468)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7c238) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7bc00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa32eaa700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 350 (Thread 0x7ffa326a9700 (LWP 16469)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7b730) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7b6c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa326a9700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 349 (Thread 0x7ffa31ea8700 (LWP 16470)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7b8d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7b6c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa31ea8700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 348 (Thread 0x7ffa316a7700 (LWP 16471)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa316a6c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7b690) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa316a7700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 347 (Thread 0x7ffa30ea6700 (LWP 16472)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7bb38) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7b500) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 ---Type <return> to continue, or q <return> to quit--- #4 0x00007ffa42448182 in start_thread (arg=0x7ffa30ea6700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 346 (Thread 0x7ffa306a5700 (LWP 16473)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7b030) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7afc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa306a5700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 345 (Thread 0x7ffa2fea4700 (LWP 16474)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7b1d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7afc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2fea4700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 344 (Thread 0x7ffa2f6a3700 (LWP 16475)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2f6a2c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7af90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa2f6a3700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 343 (Thread 0x7ffa2eea2700 (LWP 16476)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7b438) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7ae00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2eea2700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 342 (Thread 0x7ffa2e6a1700 (LWP 16477)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7a930) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7a8c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2e6a1700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 341 (Thread 0x7ffa2dea0700 (LWP 16478)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7aad0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7a8c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2dea0700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 340 (Thread 0x7ffa2d69f700 (LWP 16479)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2d69ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7a890) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa2d69f700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 339 (Thread 0x7ffa2ce9e700 (LWP 16480)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7ad38) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7a700) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2ce9e700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 338 (Thread 0x7ffa2c69d700 (LWP 16481)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7cc30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7cbc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2c69d700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 337 (Thread 0x7ffa2be9c700 (LWP 16482)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7cdd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7cbc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2be9c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 336 (Thread 0x7ffa2b69b700 (LWP 16483)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2b69ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7cb90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa2b69b700) at pthread_create.c:312 ---Type <return> to continue, or q <return> to quit--- #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 335 (Thread 0x7ffa2ae9a700 (LWP 16484)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7d038) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7ca00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2ae9a700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 334 (Thread 0x7ffa2a699700 (LWP 16485)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa46d7da30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa46d7d9c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2a699700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 333 (Thread 0x7ffa29e98700 (LWP 16486)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa46d7dbd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa46d7d9c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa29e98700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 332 (Thread 0x7ffa29697700 (LWP 16487)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa29696c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa46d7d990) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa29697700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 331 (Thread 0x7ffa28e96700 (LWP 16488)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa46d7de38) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa46d7d800) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa28e96700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 330 (Thread 0x7ffa28695700 (LWP 16489)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6de530) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa6f6de4c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa28695700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 329 (Thread 0x7ffa27e94700 (LWP 16490)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6de6d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa6f6de4c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa27e94700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 328 (Thread 0x7ffa27693700 (LWP 16491)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa27692c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa6f6de490) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa27693700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 327 (Thread 0x7ffa26e92700 (LWP 16492)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6de938) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa6f6de300) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa26e92700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 326 (Thread 0x7ffa26691700 (LWP 16493)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dc230) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa6f6dc1c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa26691700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 325 (Thread 0x7ffa25e90700 (LWP 16494)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6dc3d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa6f6dc1c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa25e90700) at pthread_create.c:312 ---Type <return> to continue, or q <return> to quit--- #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 324 (Thread 0x7ffa2568f700 (LWP 16495)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2568ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa6f6dc190) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa2568f700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 323 (Thread 0x7ffa24e8e700 (LWP 16496)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6dc638) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa6f6dc000) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa24e8e700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 322 (Thread 0x7ffa2468d700 (LWP 16497)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dd030) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa6f6dcfc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa2468d700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 321 (Thread 0x7ffa23e8c700 (LWP 16498)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6dd1d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa6f6dcfc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa23e8c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 320 (Thread 0x7ffa2368b700 (LWP 16499)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa2368ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa6f6dcf90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa2368b700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 319 (Thread 0x7ffa22e8a700 (LWP 16500)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6dd438) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa6f6dce00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa22e8a700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 318 (Thread 0x7ffa22689700 (LWP 16501)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffa6f6dde30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffa6f6dddc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa22689700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 317 (Thread 0x7ffa21e88700 (LWP 16502)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffa6f6ddfd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffa6f6dddc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa21e88700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 316 (Thread 0x7ffa21687700 (LWP 16503)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa21686c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffa6f6ddd90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa21687700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 315 (Thread 0x7ffa20e86700 (LWP 16504)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffa6f6de238) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffa6f6ddc00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa20e86700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 314 (Thread 0x7ffa20685700 (LWP 16505)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4962230) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffab49621c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa20685700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 313 (Thread 0x7ffa1fe84700 (LWP 16506)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49623d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffab49621c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1fe84700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 312 (Thread 0x7ffa1f683700 (LWP 16507)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1f682c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffab4962190) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa1f683700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 311 (Thread 0x7ffa1ee82700 (LWP 16508)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4962638) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffab4962000) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1ee82700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 310 (Thread 0x7ffa1e681700 (LWP 16509)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4964530) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffab49644c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1e681700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 309 (Thread 0x7ffa1de80700 (LWP 16510)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49646d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffab49644c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1de80700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 308 (Thread 0x7ffa1d67f700 (LWP 16511)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1d67ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffab4964490) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa1d67f700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 307 (Thread 0x7ffa1ce7e700 (LWP 16512)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4964938) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffab4964300) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1ce7e700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 306 (Thread 0x7ffa1c67d700 (LWP 16513)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffab4963030) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffab4962fc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1c67d700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 305 (Thread 0x7ffa1be7c700 (LWP 16514)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffab49631d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffab4962fc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1be7c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 304 (Thread 0x7ffa1b67b700 (LWP 16515)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1b67ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffab4962f90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa1b67b700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 303 (Thread 0x7ffa1ae7a700 (LWP 16516)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffab4963438) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffab4962e00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1ae7a700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 302 (Thread 0x7ffa1a679700 (LWP 16517)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21730) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e216c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1a679700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 301 (Thread 0x7ffa19e78700 (LWP 16518)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e218d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e216c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa19e78700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 300 (Thread 0x7ffa19677700 (LWP 16519)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa19676c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e21690) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa19677700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 299 (Thread 0x7ffa18e76700 (LWP 16520)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e21b38) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e21500) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa18e76700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 298 (Thread 0x7ffa18675700 (LWP 16521)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21e30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e21dc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa18675700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 297 (Thread 0x7ffa17e74700 (LWP 16522)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e21fd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e21dc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa17e74700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 296 (Thread 0x7ffa17673700 (LWP 16523)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa17672c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e21d90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa17673700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 295 (Thread 0x7ffa16e72700 (LWP 16524)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e22238) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e21c00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa16e72700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 294 (Thread 0x7ffa16671700 (LWP 16525)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e23a30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e239c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa16671700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 293 (Thread 0x7ffa15e70700 (LWP 16526)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e23bd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e239c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa15e70700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 292 (Thread 0x7ffa1566f700 (LWP 16527)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1566ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e23990) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa1566f700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 291 (Thread 0x7ffa14e6e700 (LWP 16528)): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23e38) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e23800) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa14e6e700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 290 (Thread 0x7ffa1466d700 (LWP 16529)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e23330) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e232c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa1466d700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 289 (Thread 0x7ffa13e6c700 (LWP 16530)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e234d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e232c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa13e6c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 288 (Thread 0x7ffa1366b700 (LWP 16531)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa1366ac60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e23290) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa1366b700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 287 (Thread 0x7ffa12e6a700 (LWP 16532)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23738) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e23100) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa12e6a700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 286 (Thread 0x7ffa12669700 (LWP 16533)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e22c30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e22bc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa12669700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 285 (Thread 0x7ffa11e68700 (LWP 16534)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e22dd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e22bc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa11e68700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 284 (Thread 0x7ffa11667700 (LWP 16535)): #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa11666c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e22b90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa11667700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 283 (Thread 0x7ffa10e66700 (LWP 16536)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e23038) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e22a00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa10e66700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 282 (Thread 0x7ffa10665700 (LWP 16537)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348d6c3 in Wait (mutex=..., this=0x7ffae2e21030) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7ffae2e20fc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007ffa4339802d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa10665700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 281 (Thread 0x7ffa0e661700 (LWP 16538)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa4348ee0c in Wait (mutex=..., this=0x7ffae2e211d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7ffae2e20fc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007ffa4339804d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0e661700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 280 (Thread 0x7ffa0de60700 (LWP 16539)): ---Type <return> to continue, or q <return> to quit--- #0 0x00007ffa40fc4cbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ffa43465b29 in poll (__timeout=-1, __nfds=1, __fds=0x7ffa0de5fc60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7ffae2e20f90) at msg/simple/Accepter.cc:216 #3 0x00007ffa42448182 in start_thread (arg=0x7ffa0de60700) at pthread_create.c:312 #4 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 279 (Thread 0x7ffa0d65f700 (LWP 16540)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa43390edf in Wait (mutex=..., this=0x7ffae2e21438) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x7ffae2e20e00) at msg/simple/SimpleMessenger.cc:212 #3 0x00007ffa43398b7d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/simple/SimpleMessenger.h:196 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0d65f700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 278 (Thread 0x7ffa0c252700 (LWP 16561)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434d62aa in Wait (mutex=..., this=0x7ffab4949228) at ./common/Cond.h:55 #2 Pipe::reader (this=0x7ffab4945000) at msg/simple/Pipe.cc:1512 #3 0x00007ffa434df75d in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0c252700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 277 (Thread 0x7ffa0b459700 (LWP 16569)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffab4949228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffab4945000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0b459700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 276 (Thread 0x7ffa0a74c700 (LWP 16583)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434d62aa in Wait (mutex=..., this=0x7ffa6fabb228) at ./common/Cond.h:55 #2 Pipe::reader (this=0x7ffa6fab7000) at msg/simple/Pipe.cc:1512 #3 0x00007ffa434df75d in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa0a74c700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 275 (Thread 0x7ffa09a3f700 (LWP 16609)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa55a78228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffa55a74000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa09a3f700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 274 (Thread 0x7ffa09136700 (LWP 16621)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa6946d228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffa69469000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa09136700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 273 (Thread 0x7ffa09035700 (LWP 16622)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa551f0228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffa551ec000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa09035700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 272 (Thread 0x7ffa09439700 (LWP 16643)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa69383228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffa6937f000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa09439700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 271 (Thread 0x7ffa07e23700 (LWP 16647)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007ffa434dc219 in Wait (mutex=..., this=0x7ffa6fabb228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7ffa6fab7000) at msg/simple/Pipe.cc:1845 #3 0x00007ffa434df77d in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007ffa42448182 in start_thread (arg=0x7ffa07e23700) at pthread_create.c:312 #5 0x00007ffa40fd200d in clone () at ../sysdeps/unix/sys
Updated by Haomai Wang over 8 years ago
Hmm, it seemed like a rare case, I can't reproduce this in local env.
Updated by Sage Weil over 8 years ago
I think the test is buggy. It randomly does drop_connection(), which will mark_down() one end of a stateful lossless_peer connection. In reality the other end will also mark_down...
The actual race is this:
- A queues messages to B
- fault
- A and B race connecting to each other
- A sends WAIT and waits for B to connect to it
- B does mark_downwhich means B never connects but A is waiting forever for it to.
We could work around this by making the connection race WAIT state retry periodically, but in reality this shouldn't happen, I think!
Updated by Sage Weil over 8 years ago
2015-08-25 14:43:32.950307 7f33c3e24700 2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault (32) Broken pipe 2015-08-25 14:43:32.950319 7f33c3e24700 0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault 2015-08-25 14:43:32.950326 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer: state = connecting policy.server=0 2015-08-25 14:43:32.950334 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect 7 2015-08-25 14:43:32.950367 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connecting to 127.0.0.1:16820/29166 2015-08-25 14:43:32.950721 7f33c3e24700 2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect couldn't read banner, (32) Broken pipe 2015-08-25 14:43:32.950763 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580). sleep for 0.1 2015-08-25 14:43:32.951018 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept peer addr is 127.0.0.1:16820/29166 2015-08-25 14:43:32.951035 7f33c341a700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept got peer connect_seq 7 global_seq 20 2015-08-25 14:43:32.951044 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept of host_type 8, policy.lossy=0 policy.server=0 policy.standby=1 policy.resetcheck=1 2015-08-25 14:43:32.951049 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept my proto 24, their proto 24 2015-08-25 14:43:32.951053 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept: setting up session_security. 2015-08-25 14:43:32.951059 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept existing 0x7f34148fb000.gseq 17 <= 20, looks ok 2015-08-25 14:43:32.951064 7f33c341a700 0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connect_seq 7 vs existing 7 state connecting 2015-08-25 14:43:32.951070 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connection race, existing 0x7f34148fb000.cseq 7 == 7, sending WAIT 2015-08-25 14:43:32.967993 7f3403770780 1 -- 127.0.0.1:16801/29147 mark_down 0x7f3414a1c580 -- 0x7f34148fb000 2015-08-25 14:43:32.968001 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).unregister_pipe 2015-08-25 14:43:32.968010 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).stop 2015-08-25 14:43:33.050859 7f33c3e24700 3 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect fault, but state = closed != connecting, stopping 2015-08-25 14:43:33.050893 7f33c3e24700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer finishing 2015-08-25 14:43:33.050918 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer done
Updated by Sage Weil over 8 years ago
- Status changed from New to 4
2015-08-25 14:43:32.950307 7f33c3e24700 2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault (32) Broken pipe 2015-08-25 14:43:32.950319 7f33c3e24700 0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).fault 2015-08-25 14:43:32.950326 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer: state = connecting policy.server=0 2015-08-25 14:43:32.950334 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=165 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect 7 2015-08-25 14:43:32.950367 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connecting to 127.0.0.1:16820/29166 2015-08-25 14:43:32.950721 7f33c3e24700 2 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect couldn't read banner, (32) Broken pipe 2015-08-25 14:43:32.950763 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580). sleep for 0.1 2015-08-25 14:43:32.951018 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept peer addr is 127.0.0.1:16820/29166 2015-08-25 14:43:32.951035 7f33c341a700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept got peer connect_seq 7 global_seq 20 2015-08-25 14:43:32.951044 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept of host_type 8, policy.lossy=0 policy.server=0 policy.standby=1 policy.resetcheck=1 2015-08-25 14:43:32.951049 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept my proto 24, their proto 24 2015-08-25 14:43:32.951053 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept: setting up session_security. 2015-08-25 14:43:32.951059 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept existing 0x7f34148fb000.gseq 17 <= 20, looks ok 2015-08-25 14:43:32.951064 7f33c341a700 0 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connect_seq 7 vs existing 7 state connecting 2015-08-25 14:43:32.951070 7f33c341a700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f3415e74000 sd=137 :16801 s=0 pgs=0 cs=0 l=0 c=0x7f3414a1dfa0).accept connection race, existing 0x7f34148fb000.cseq 7 == 7, sending WAIT 2015-08-25 14:43:32.967993 7f3403770780 1 -- 127.0.0.1:16801/29147 mark_down 0x7f3414a1c580 -- 0x7f34148fb000 2015-08-25 14:43:32.968001 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).unregister_pipe 2015-08-25 14:43:32.968010 7f3403770780 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=1 pgs=17 cs=7 l=0 c=0x7f3414a1c580).stop 2015-08-25 14:43:33.050859 7f33c3e24700 3 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).connect fault, but state = closed != connecting, stopping 2015-08-25 14:43:33.050893 7f33c3e24700 20 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer finishing 2015-08-25 14:43:33.050918 7f33c3e24700 10 -- 127.0.0.1:16801/29147 >> 127.0.0.1:16820/29166 pipe(0x7f34148fb000 sd=81 :16801 s=4 pgs=17 cs=7 l=0 c=0x7f3414a1c580).writer done
Updated by Greg Farnum about 5 years ago
- Project changed from Ceph to Messengers
- Category deleted (
msgr)
Actions