Actions
Bug #8822
closedosd: hang on shutdown, spinlocks
Added by Sage Weil almost 10 years ago. Updated over 9 years ago.
Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-07-13_02:30:04-rados-next-testing-basic-plana/357857
put a copy of osd.3's log in the archive dir
Updated by Sage Weil almost 10 years ago
- Description updated (diff)
- Category set to OSD
(gdb) bt #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00000000052b6318 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00000000052b6084 in __pthread_mutex_cond_lock (mutex=0x2e08e3c8) at ../nptl/pthread_mutex_lock.c:79 #3 0x00000000052b04a4 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:259 #4 0x0000000000b1e678 in Wait (mutex=..., this=0x2e08e490) at ./common/Cond.h:55 #5 Pipe::writer (this=0x2e08e280) at msg/Pipe.cc:1766 #6 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #7 0x00000000052ac182 in start_thread (arg=0x2e94a700) at pthread_create.c:312 #8 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb) thr app all bt Thread 118 (Thread 24423): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e090c90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e090a80) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2b23b700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 117 (Thread 23711): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4112c90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4112a80) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2d45b700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 116 (Thread 24422): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2c047b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e090a80) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e090a80, buf=buf@entry=0x2c047cbf "\377\002", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2e090a80) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2c048700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 115 (Thread 24320): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e251890) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e251680) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2ea4b700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 114 (Thread 24319): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00000000052b6318 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00000000052b6084 in __pthread_mutex_cond_lock (mutex=0x2e08e3c8) at ../nptl/pthread_mutex_lock.c:79 #3 0x00000000052b04a4 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:259 #4 0x0000000000b1e678 in Wait (mutex=..., this=0x2e08e490) at ./common/Cond.h:55 #5 Pipe::writer (this=0x2e08e280) at msg/Pipe.cc:1766 #6 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #7 0x00000000052ac182 in start_thread (arg=0x2e94a700) at pthread_create.c:312 #8 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 113 (Thread 24318): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2eb4b480) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e251680) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e251680, buf=buf@entry=0x2eb4b8d0 "", len=len@entry=21) at msg/Pipe.cc:2294 #4 0x0000000000b12dc3 in Pipe::read_message (this=this@entry=0x2e251680, pm=pm@entry=0x2eb4bcf0, auth_handler=auth_handler@entry=0x2d753f20) at msg/Pipe.cc:1969 #5 0x0000000000b266ab in Pipe::reader (this=0x2e251680) at msg/Pipe.cc:1508 #6 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #7 0x00000000052ac182 in start_thread (arg=0x2eb4c700) at pthread_create.c:312 #8 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 112 (Thread 24195): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2ad35b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e090800) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e090800, buf=buf@entry=0x2ad35cbf "\377t\024", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2e090800) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2ad36700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 111 (Thread 23860): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2afb10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2af900) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2f352700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 110 (Thread 24087): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2aec10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2aea00) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2a232700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 109 (Thread 24086): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4112790) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4112580) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2a131700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 108 (Thread 23913): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e08e210) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e08e000) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2d159700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 107 (Thread 23912): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2f3e8790) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2f3e8580) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2ce56700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 106 (Thread 23911): #0 0x00000000052b37eb in __libc_recv (fd=95, buf=buf@entry=0x2d057750, n=n@entry=53, flags=86718443, flags@entry=64) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33 #1 0x0000000000b11762 in recv (__flags=64, __n=53, __buf=0x2d057750, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/socket2.h:44 #2 Pipe::tcp_read_nonblocking (this=this@entry=0x2e08e000, buf=buf@entry=0x2d057750 "\263", len=len@entry=53) at msg/Pipe.cc:2340 #3 0x0000000000b119ad in Pipe::tcp_read (this=this@entry=0x2e08e000, buf=buf@entry=0x2d057750 "\263", len=len@entry=53) at msg/Pipe.cc:2297 #4 0x0000000000b11e8f in Pipe::read_message (this=this@entry=0x2e08e000, pm=pm@entry=0x2d057cf0, auth_handler=auth_handler@entry=0x29e7b360) at msg/Pipe.cc:1826 #5 0x0000000000b266ab in Pipe::reader (this=0x2e08e000) at msg/Pipe.cc:1508 #6 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #7 0x00000000052ac182 in start_thread (arg=0x2d058700) at pthread_create.c:312 #8 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 105 (Thread 23910): #0 0x00000000052b37eb in __libc_recv (fd=94, buf=buf@entry=0x2cf56750, n=n@entry=53, flags=86718443, flags@entry=64) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33 #1 0x0000000000b11762 in recv (__flags=64, __n=53, __buf=0x2cf56750, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/socket2.h:44 #2 Pipe::tcp_read_nonblocking (this=this@entry=0x2f3e8580, buf=buf@entry=0x2cf56750 "\263", len=len@entry=53) at msg/Pipe.cc:2340 #3 0x0000000000b119ad in Pipe::tcp_read (this=this@entry=0x2f3e8580, buf=buf@entry=0x2cf56750 "\263", len=len@entry=53) at msg/Pipe.cc:2297 #4 0x0000000000b11e8f in Pipe::read_message (this=this@entry=0x2f3e8580, pm=pm@entry=0x2cf56cf0, auth_handler=auth_handler@entry=0x29e7b300) at msg/Pipe.cc:1826 #5 0x0000000000b266ab in Pipe::reader (this=0x2f3e8580) at msg/Pipe.cc:1508 #6 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #7 0x00000000052ac182 in start_thread (arg=0x2cf57700) at pthread_create.c:312 #8 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 104 (Thread 23996): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2b0010) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2afe00) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2b840700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 103 (Thread 24230): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2bf46b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2b2afb80) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2b2afb80, buf=buf@entry=0x2bf46cbf "\377\220", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2b2afb80) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2bf47700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 102 (Thread 24220): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4112a10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4112800) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 ---Type <return> to continue, or q <return> to quit--- #4 0x00000000052ac182 in start_thread (arg=0x2ca52700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 101 (Thread 24219): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2f3e7b10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2f3e7900) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2cd55700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 100 (Thread 24317): #0 pthread_cond_broadcast@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S:98 #1 0x0000000000b2676e in Signal (this=0x2e08e490) at ./common/Cond.h:91 #2 Pipe::reader (this=0x2e08e280) at msg/Pipe.cc:1547 #3 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #4 0x00000000052ac182 in start_thread (arg=0x2e849700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 99 (Thread 23730): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4113190) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4112f80) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2bc44700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 98 (Thread 24217): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2cb52b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4112800) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4112800, buf=buf@entry=0x2cb52cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4112800) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2cb53700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 97 (Thread 24194): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2b0510) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2b0300) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2f050700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 96 (Thread 23391): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4113b90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4113980) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c54d700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 95 (Thread 23389): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4113690) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4113480) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c44c700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 94 (Thread 23387): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2c34ab10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4113980) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4113980, buf=buf@entry=0x2c34acbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4113980) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2c34b700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 93 (Thread 23386): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2c249b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4113480) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4113480, buf=buf@entry=0x2c249cbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 ---Type <return> to continue, or q <return> to quit--- #4 0x0000000000b25b8f in Pipe::reader (this=0x4113480) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2c24a700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 92 (Thread 24289): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2b139b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4111900) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4111900, buf=buf@entry=0x2b139cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4111900) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2b13a700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 91 (Thread 24288): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2c148b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4112300) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4112300, buf=buf@entry=0x2c148cbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4112300) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2c149700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 90 (Thread 24229): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2ae210) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2ae000) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0xbd85700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 89 (Thread 24003): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2af37b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2b2afe00) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2b2afe00, buf=buf@entry=0x2af37cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2b2afe00) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2af38700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 88 (Thread 24034): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2b63db10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e252d00) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e252d00, buf=buf@entry=0x2b63dcbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2e252d00) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2b63e700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 87 (Thread 23713): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2ed4db10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4112a80) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4112a80, buf=buf@entry=0x2ed4dcbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4112a80) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2ed4e700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 86 (Thread 23610): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4111110) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4110f00) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2bb43700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 85 (Thread 24033): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2ba41b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e252a80) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e252a80, buf=buf@entry=0x2ba41cbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 ---Type <return> to continue, or q <return> to quit--- #4 0x0000000000b25b8f in Pipe::reader (this=0x2e252a80) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2ba42700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 84 (Thread 24031): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e252c90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e252a80) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c74f700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 83 (Thread 23613): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2d359b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4110f00) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4110f00, buf=buf@entry=0x2d359cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4110f00) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2d35a700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 82 (Thread 24215): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2cc53b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2f3e7900) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2f3e7900, buf=buf@entry=0x2cc53cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2f3e7900) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2cc54700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 81 (Thread 24032): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e252f10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e252d00) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2b039700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 80 (Thread 23710): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e090790) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e090580) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c64e700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 79 (Thread 23712): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2b53cb10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2e090580) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2e090580, buf=buf@entry=0x2b53ccbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2e090580) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2b53d700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 78 (Thread 24282): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4112510) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4112300) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c951700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 77 (Thread 24283): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4111b10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4111900) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x29e2f700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 76 (Thread 23997): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2ae990) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2ae780) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2a030700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 75 (Thread 24001): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2be45b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2b2ae780) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2b2ae780, buf=buf@entry=0x2be45cbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2b2ae780) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2be46700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 74 (Thread 23612): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2b73eb10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4113c00) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4113c00, buf=buf@entry=0x2b73ecbf "\377", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4113c00) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2b73f700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 73 (Thread 23611): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x4113e10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4113c00) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2b43c700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 72 (Thread 23348): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000097b380 in poll (__timeout=-1, __nfds=4, __fds=0x2ac34d10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #2 SignalHandler::entry (this=0xa3c8780) at global/signal_handler.cc:213 #3 0x00000000052ac182 in start_thread (arg=0x2ac35700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 71 (Thread 24085): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2e747b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2b2aea00) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2b2aea00, buf=buf@entry=0x2e747cbf "\377/", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2b2aea00) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2e748700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 70 (Thread 24084): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2ae36b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4112580) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4112580, buf=buf@entry=0x2ae36cbf "\377\002", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4112580) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2ae37700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 69 (Thread 24415): #0 pthread_cond_broadcast@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S:98 #1 0x0000000000b2676e in Signal (this=0x2b2ae210) at ./common/Cond.h:91 #2 Pipe::reader (this=0x2b2ae000) at msg/Pipe.cc:1547 #3 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #4 0x00000000052ac182 in start_thread (arg=0x2a434700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 68 (Thread 24193): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x2a332b10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x2b2b0300) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x2b2b0300, buf=buf@entry=0x2a332cbf "\377\020\001", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x2b2b0300) at msg/Pipe.cc:1446 ---Type <return> to continue, or q <return> to quit--- #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2a333700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 67 (Thread 24232): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2b2afd90) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2b2afb80) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2c850700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 66 (Thread 24246): #0 0x00000000052b37eb in __libc_recv (fd=150, buf=buf@entry=0x2ec4cce0, n=n@entry=8, flags=86718443, flags@entry=64) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33 #1 0x0000000000b11762 in recv (__flags=64, __n=8, __buf=0x2ec4cce0, __fd=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/socket2.h:44 #2 Pipe::tcp_read_nonblocking (this=this@entry=0x2b2af900, buf=buf@entry=0x2ec4cce0 "\223\n", len=len@entry=8) at msg/Pipe.cc:2340 #3 0x0000000000b119ad in Pipe::tcp_read (this=this@entry=0x2b2af900, buf=buf@entry=0x2ec4cce0 "\223\n", len=len@entry=8) at msg/Pipe.cc:2297 #4 0x0000000000b264f0 in Pipe::reader (this=0x2b2af900) at msg/Pipe.cc:1494 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x2ec4d700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 65 (Thread 24196): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b1e678 in Wait (mutex=..., this=0x2e090a10) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2e090800) at msg/Pipe.cc:1766 #3 0x0000000000b2a17d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61 #4 0x00000000052ac182 in start_thread (arg=0x2e45c700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 64 (Thread 23326): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000066ba3e in Cond::Wait (this=0xa3bfec0, mutex=...) at ./common/Cond.h:55 #2 0x000000000060a54c in OSDService::agent_entry (this=0xa3bf6a0) at osd/OSD.cc:523 #3 0x000000000067456d in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:561 #4 0x00000000052ac182 in start_thread (arg=0x29c2d700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 63 (Thread 23325): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000a4a214 in Wait (mutex=..., this=0xa3c0010) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0xa3c0000) at common/Timer.cc:112 #3 0x0000000000a4ae6d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00000000052ac182 in start_thread (arg=0x2942c700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 62 (Thread 23324): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000a4a214 in Wait (mutex=..., this=0xa3c05b0) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0xa3c05a0) at common/Timer.cc:112 #3 0x0000000000a4ae6d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00000000052ac182 in start_thread (arg=0x28c2b700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 61 (Thread 23731): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000b116df in poll (__timeout=<optimized out>, __nfds=1, __fds=0x29d2db10) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x4112f80) at msg/Pipe.cc:2321 #3 0x0000000000b11998 in Pipe::tcp_read (this=this@entry=0x4112f80, buf=buf@entry=0x29d2dcbf "\377\004", len=len@entry=1) at msg/Pipe.cc:2294 #4 0x0000000000b25b8f in Pipe::reader (this=0x4112f80) at msg/Pipe.cc:1446 #5 0x0000000000b2a19d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49 #6 0x00000000052ac182 in start_thread (arg=0x29d2e700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 60 (Thread 23322): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000a4a62f in WaitUntil (when=..., mutex=..., this=0xa3c0130) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0xa3c0120) at common/Timer.cc:114 #3 0x0000000000a4ae6d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00000000052ac182 in start_thread (arg=0x2842a700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 59 (Thread 23321): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000099e2d8 in Wait (mutex=..., this=0xa3c0450) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0xa3c03f0) at common/Finisher.cc:78 ---Type <return> to continue, or q <return> to quit--- #3 0x00000000052ac182 in start_thread (arg=0x27c29700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 58 (Thread 23320): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000099e2d8 in Wait (mutex=..., this=0xa3c0850) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0xa3c07f0) at common/Finisher.cc:78 #3 0x00000000052ac182 in start_thread (arg=0x27428700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 57 (Thread 23319): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000062d051 in WaitUntil (when=..., mutex=..., this=0xa3becf8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0xa3becf8) at ./common/Cond.h:79 #3 OSD::heartbeat_entry (this=0xa3be000) at osd/OSD.cc:3163 #4 0x0000000000694c8d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:1286 #5 0x00000000052ac182 in start_thread (arg=0x26c27700) at pthread_create.c:312 #6 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 56 (Thread 23318): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000a5181c in WaitUntil (when=..., mutex=..., this=0xa3beac8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0xa3beac8) at common/Cond.h:79 #3 ThreadPool::worker (this=0xa3bea50, wt=0xa477000) at common/WorkQueue.cc:146 #4 0x0000000000a52d50 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00000000052ac182 in start_thread (arg=0x26426700) at pthread_create.c:312 #6 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 55 (Thread 23317): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000a5181c in WaitUntil (when=..., mutex=..., this=0xa3be938) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0xa3be938) at common/Cond.h:79 #3 ThreadPool::worker (this=0xa3be8c0, wt=0xa477060) at common/WorkQueue.cc:146 #4 0x0000000000a52d50 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00000000052ac182 in start_thread (arg=0x25c25700) at pthread_create.c:312 #6 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 54 (Thread 23316): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000a5181c in WaitUntil (when=..., mutex=..., this=0xa3be7a8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0xa3be7a8) at common/Cond.h:79 #3 ThreadPool::worker (this=0xa3be730, wt=0xa477120) at common/WorkQueue.cc:146 #4 0x0000000000a52d50 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #5 0x00000000052ac182 in start_thread (arg=0x25424700) at pthread_create.c:312 #6 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 53 (Thread 23315): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cca18) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cca18) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa4775a0) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x24c23700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 52 (Thread 23314): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc398) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc398) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa477510) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x24422700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 51 (Thread 23313): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc538) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc538) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa4774b0) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x23c21700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 50 (Thread 23312): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc058) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc058) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa4774e0) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x23420700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 49 (Thread 23311): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc1f8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc1f8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa477450) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x22c1f700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 48 (Thread 23309): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc398) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc398) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa4323c0) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x21c1d700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 47 (Thread 23308): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc538) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc538) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa432240) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x2141c700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 46 (Thread 23307): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc058) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc058) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa432270) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x20c1b700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 45 (Thread 23306): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cc1f8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cc1f8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa4322d0) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x2041a700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 44 (Thread 23305): #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33 #1 0x000000000063acaa in ceph_spin_lock (l=0xa3be460) at ./include/Spinlock.h:45 #2 lock (this=0xa3be460) at ./include/Spinlock.h:94 #3 Locker (s=..., this=<synthetic pointer>) at ./include/Spinlock.h:105 #4 is_active (this=0xa3be000) at osd/OSD.h:1083 #5 OSD::dispatch_context (this=this@entry=0xa3be000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x1fc18b70) at osd/OSD.cc:6896 #6 0x000000000064b047 in OSD::process_peering_events (this=0xa3be000, pgs=..., handle=...) at osd/OSD.cc:8276 #7 0x00000000006a0048 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1499 #8 0x0000000000a51c61 in ThreadPool::worker (this=0xa3be468, wt=0xa4320f0) at common/WorkQueue.cc:128 #9 0x0000000000a52d50 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #10 0x00000000052ac182 in start_thread (arg=0x1fc19700) at pthread_create.c:312 #11 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 43 (Thread 23304): #0 is_active (this=0xa3be000) at osd/OSD.h:1084 ---Type <return> to continue, or q <return> to quit--- #1 OSD::dispatch_context (this=this@entry=0xa3be000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x1f417b70) at osd/OSD.cc:6896 #2 0x000000000064b047 in OSD::process_peering_events (this=0xa3be000, pgs=..., handle=...) at osd/OSD.cc:8276 #3 0x00000000006a0048 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1499 #4 0x0000000000a51c61 in ThreadPool::worker (this=0xa3be468, wt=0xa432090) at common/WorkQueue.cc:128 #5 0x0000000000a52d50 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #6 0x00000000052ac182 in start_thread (arg=0x1f418700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 42 (Thread 23303): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000099e2d8 in Wait (mutex=..., this=0xfff0006e0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0xfff000680) at common/Finisher.cc:78 #3 0x00000000052ac182 in start_thread (arg=0x1ec17700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 41 (Thread 23302): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000a4a62f in WaitUntil (when=..., mutex=..., this=0xfff0005c8) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0xfff0005b8) at common/Timer.cc:114 #3 0x0000000000a4ae6d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00000000052ac182 in start_thread (arg=0x1e416700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (Thread 23310): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000635888 in WaitUntil (when=..., mutex=..., this=0x40cca18) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x40cca18) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0xa3befa8, thread_index=<optimized out>, hb=0xa477480) at osd/OSD.cc:8043 #4 0x0000000000a50ade in ShardedThreadPool::shardedthreadpool_worker (this=0xa3be5f8, thread_index=<optimized out>) at common/WorkQueue.cc:320 #5 0x0000000000a52d30 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:504 #6 0x00000000052ac182 in start_thread (arg=0x2241e700) at pthread_create.c:312 #7 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (Thread 23300): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aeb3e9 in poll (__timeout=-1, __nfds=1, __fds=0x1dc14c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x40915d0) at msg/Accepter.cc:196 #3 0x00000000052ac182 in start_thread (arg=0x1dc15700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (Thread 23299): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4091810) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4091608) at msg/DispatchQueue.cc:114 #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x1d414700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (Thread 23298): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4091670) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4091608) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00000000052ac182 in start_thread (arg=0x1cc13700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (Thread 23297): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aeb3e9 in poll (__timeout=-1, __nfds=1, __fds=0x1c411c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x40923d0) at msg/Accepter.cc:196 #3 0x00000000052ac182 in start_thread (arg=0x1c412700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (Thread 23296): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4092610) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4092408) at msg/DispatchQueue.cc:114 #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x1bc11700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 23295): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4092470) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4092408) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 ---Type <return> to continue, or q <return> to quit--- #4 0x00000000052ac182 in start_thread (arg=0x1b410700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 23294): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aeb3e9 in poll (__timeout=-1, __nfds=1, __fds=0x1ac0ec60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x4091cd0) at msg/Accepter.cc:196 #3 0x00000000052ac182 in start_thread (arg=0x1ac0f700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 23293): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4091f10) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4091d08) at msg/DispatchQueue.cc:114 #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x1a40e700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 23292): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4091d70) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4091d08) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00000000052ac182 in start_thread (arg=0x19c0d700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 23291): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4090a10) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4090808) at msg/DispatchQueue.cc:114 #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x1940c700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 23290): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4090870) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4090808) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00000000052ac182 in start_thread (arg=0x18c0b700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 23289): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aeb3e9 in poll (__timeout=-1, __nfds=1, __fds=0x18409c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x4090ed0) at msg/Accepter.cc:196 #3 0x00000000052ac182 in start_thread (arg=0x1840a700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 23288): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4091110) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4090f08) at msg/DispatchQueue.cc:114 #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x17c09700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (Thread 23287): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4090f70) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4090f08) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00000000052ac182 in start_thread (arg=0x17408700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 23286): #0 0x0000000006f3bfbd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aeb3e9 in poll (__timeout=-1, __nfds=1, __fds=0x16c06c60) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x40900d0) at msg/Accepter.cc:196 #3 0x00000000052ac182 in start_thread (arg=0x16c07700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (Thread 23285): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000b0184e in Wait (mutex=..., this=0x4090310) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x4090108) at msg/DispatchQueue.cc:114 ---Type <return> to continue, or q <return> to quit--- #3 0x0000000000a4034d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/DispatchQueue.h:118 #4 0x00000000052ac182 in start_thread (arg=0x16406700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 23284): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000afeaa3 in Wait (mutex=..., this=0x4090170) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x4090108) at msg/DispatchQueue.cc:196 #3 0x0000000000a4032d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00000000052ac182 in start_thread (arg=0x15c05700) at pthread_create.c:312 #5 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 23277): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000099e2d8 in Wait (mutex=..., this=0x4094558) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x40944f8) at common/Finisher.cc:78 #3 0x00000000052ac182 in start_thread (arg=0x12db1700) at pthread_create.c:312 #4 0x0000000006f4930d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:1
Updated by Sage Weil almost 10 years ago
saw this again, ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-07-27_02:30:01-rados-next-testing-basic-plana/380444
all threads in threads.txt
interesting bits:
Thread 18 (Thread 6390): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00000000052b6318 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00000000052b6084 in __pthread_mutex_cond_lock (mutex=0x4096430) at ../nptl/pthread_mutex_lock.c:79 #3 0x00000000052b04a4 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:259 #4 0x00000000008a8490 in Wait (mutex=..., this=0x4096478) at ./common/Cond.h:55 #5 IndexManager::get_index (this=this@entry=0x4096420, c=..., path=path@entry=0x11694040 "/var/lib/ceph/osd/ceph-4/current/meta", index=index@entry=0x116950c0) at os/IndexManager.cc:132 ---Type <return> to continue, or q <return> to quit--- #6 0x000000000085d47b in FileStore::get_index (this=this@entry=0x4096000, cid=..., index=index@entry=0x116950c0) at os/FileStore.cc:152 #7 0x000000000086f952 in FileStore::lfn_find (this=this@entry=0x4096000, cid=..., oid=..., path=path@entry=0x11695140) at os/FileStore.cc:170 #8 0x0000000000871a7e in FileStore::_omap_setkeys (this=this@entry=0x4096000, cid=..., hoid=..., aset=..., spos=...) at os/FileStore.cc:4670 #9 0x0000000000890b6f in FileStore::_do_transaction (this=this@entry=0x4096000, t=..., op_seq=op_seq@entry=467, trans_num=trans_num@entry=0, handle=handle@entry=0x11695b70) at os/FileStore.cc:2416 #10 0x0000000000893fdc in FileStore::_do_transactions (this=this@entry=0x4096000, tls=..., op_seq=467, handle=handle@entry=0x11695b70) at os/FileStore.cc:1886 #11 0x0000000000894160 in FileStore::_do_op (this=0x4096000, osr=<optimized out>, handle=...) at os/FileStore.cc:1696 #12 0x0000000000a542a1 in ThreadPool::worker (this=0x4096dd8, wt=0x40fec30) at common/WorkQueue.cc:128 #13 0x0000000000a55390 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #14 0x00000000052ac182 in start_thread (arg=0x11696700) at pthread_create.c:312 #15 0x0000000006dc330d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (Thread 6432): #0 pthread_spin_unlock () at ../nptl/sysdeps/x86_64/pthread_spin_unlock.S:23 #1 0x0000000000619f25 in ceph_spin_unlock (l=0x4106460) at ./include/Spinlock.h:50 #2 unlock (this=0x4106460) at ./include/Spinlock.h:98 #3 ~Locker (this=<synthetic pointer>, __in_chrg=<optimized out>) at ./include/Spinlock.h:108 #4 is_active (this=0x4106000) at osd/OSD.h:1084 #5 OSD::require_same_or_newer_map (this=this@entry=0x4106000, op=..., epoch=<optimized out>) at osd/OSD.cc:6596 #6 0x000000000063c6d0 in OSD::handle_pg_query (this=0x4106000, op=...) at osd/OSD.cc:7333 #7 0x000000000064921a in OSD::dispatch_op (this=0x4106000, op=...) at osd/OSD.cc:5193 #8 0x00000000006548ce in OSD::_dispatch (this=this@entry=0x4106000, m=m@entry=0x29bd8960) at osd/OSD.cc:5356 #9 0x00000000006551af in OSD::ms_dispatch (this=0x4106000, m=0x29bd8960) at osd/OSD.cc:4975 ---Type <return> to continue, or q <return> to quit--- #10 0x0000000000b01309 in ms_deliver_dispatch (m=0x29bd8960, this=0x4092e00) at msg/Messenger.h:636 #11 DispatchQueue::entry (this=0x4092f08) at msg/DispatchQueue.cc:185 #12 0x0000000000a4296d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #13 0x00000000052ac182 in start_thread (arg=0x16fe1700) at pthread_create.c:312 #14 0x0000000006dc330d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 41 (Thread 6466): #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33 #1 0x000000000062fcba in ceph_spin_lock (l=0x4106460) at ./include/Spinlock.h:45 #2 lock (this=0x4106460) at ./include/Spinlock.h:94 #3 Locker (s=..., this=<synthetic pointer>) at ./include/Spinlock.h:105 #4 is_active (this=0x4106000) at osd/OSD.h:1083 #5 OSD::dispatch_context (this=this@entry=0x4106000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x1eff0b70) at osd/OSD.cc:6904 #6 0x00000000006401b7 in OSD::process_peering_events (this=0x4106000, pgs=..., handle=...) at osd/OSD.cc:8284 #7 0x0000000000695488 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1499 #8 0x0000000000a542a1 in ThreadPool::worker (this=0x4106468, wt=0x4100d60) at common/WorkQueue.cc:128 #9 0x0000000000a55390 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #10 0x00000000052ac182 in start_thread (arg=0x1eff1700) at pthread_create.c:312 #11 0x0000000006dc330d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 6391): #0 pthread_cond_broadcast@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S:98 #1 0x00000000008a6a1c in Signal (this=0x4096478) at ./common/Cond.h:91 #2 IndexManager::put_index (this=0x4096420, c=...) at os/IndexManager.cc:68 #3 0x00000000008a94dc in operator() (index=0x4058e20, this=0x2d768748) at os/IndexManager.h:69 #4 std::tr1::_Sp_counted_base_impl<HashIndex*, IndexManager::RemoveOnDelete, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x2d768730) at /usr/include/c++/4.8/tr1/shared_ptr.h:257 #5 0x00000000008dc3b2 in _M_release (this=0x2d768730) at /usr/include/c++/4.8/tr1/shared_ptr.h:141 #6 ~__shared_count (this=0x2e00daf0, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:341 #7 ~__shared_ptr (this=0x2e00dae8, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:541 #8 ~shared_ptr (this=0x2e00dae8, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:985 #9 ~Path (this=0x2e00dae0, __in_chrg=<optimized out>) at os/CollectionIndex.h:41 #10 operator() (this=<optimized out>, __p=0x2e00dae0) at /usr/include/c++/4.8/tr1/shared_ptr.h:285 #11 std::tr1::_Sp_counted_base_impl<CollectionIndex::Path*, std::tr1::_Sp_deleter<CollectionIndex::Path>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:257 #12 0x0000000000668ec9 in std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x2e00d9c0) at /usr/include/c++/4.8/tr1/shared_ptr.h:141 #13 0x0000000000871ad0 in ~__shared_count (this=0x11e96148, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:341 #14 ~__shared_ptr (this=0x11e96140, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:541 #15 ~shared_ptr (this=0x11e96140, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:985 #16 FileStore::_omap_setkeys (this=this@entry=0x4096000, cid=..., hoid=..., aset=..., spos=...) at os/FileStore.cc:4673 #17 0x0000000000890b6f in FileStore::_do_transaction (this=this@entry=0x4096000, t=..., op_seq=op_seq@entry=466, trans_num=trans_num@entry=0, handle=handle@entry=0x11e96b70) at os/FileStore.cc:2416 #18 0x0000000000893fdc in FileStore::_do_transactions (this=this@entry=0x4096000, tls=..., op_seq=466, handle=handle@entry=0x11e96b70) at os/FileStore.cc:1886 #19 0x0000000000894160 in FileStore::_do_op (this=0x4096000, osr=<optimized out>, handle=...) at os/FileStore.cc:1696 #20 0x0000000000a542a1 in ThreadPool::worker (this=0x4096dd8, wt=0x40febd0) at common/WorkQueue.cc:128 #21 0x0000000000a55390 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #22 0x00000000052ac182 in start_thread (arg=0x11e97700) at pthread_create.c:312 #23 0x0000000006dc330d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Updated by Sage Weil over 9 years ago
- Status changed from New to Can't reproduce
Updated by Sage Weil over 9 years ago
- Status changed from Can't reproduce to 12
happened again:
Thread 43 (Thread 21954): #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33 #1 0x000000000068c11a in ceph_spin_lock (l=0x41264a8) at ./include/Spinlock.h:45 #2 lock (this=0x41264a8) at ./include/Spinlock.h:94 #3 Locker (s=..., this=<synthetic pointer>) at ./include/Spinlock.h:105 #4 is_active (this=0x4126000) at osd/OSD.h:1074 #5 OSD::dispatch_context (this=this@entry=0x4126000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x21b38a70) at osd/OSD.cc:7129 #6 0x0000000000698977 in OSD::process_peering_events (this=0x4126000, pgs=..., handle=...) at osd/OSD.cc:8467 #7 0x00000000006edfb8 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1595 #8 0x0000000000b70ee6 in ThreadPool::worker (this=0x41264b0, wt=0xc412090) at common/WorkQueue.cc:128 #9 0x0000000000b71f90 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318 #10 0x0000000005d72182 in start_thread (arg=0x21b39700) at pthread_create.c:312 #11 0x000000000784438d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 21938): #0 pthread_spin_unlock () at ../nptl/sysdeps/x86_64/pthread_spin_unlock.S:23 #1 0x00000000006597de in ceph_spin_unlock (l=0x41264a8) at ./include/Spinlock.h:50 #2 unlock (this=0x41264a8) at ./include/Spinlock.h:98 #3 ~Locker (this=<synthetic pointer>, __in_chrg=<optimized out>) at ./include/Spinlock.h:108 #4 is_active (this=0x4126000) at osd/OSD.h:1075 #5 OSD::require_self_aliveness (this=this@entry=0x4126000, op=..., epoch=epoch@entry=2071) at osd/OSD.cc:6742 #6 0x00000000006776fe in OSD::require_same_or_newer_map (this=this@entry=0x4126000, op=..., epoch=2071, is_fast_dispatch=is_fast_dispatch@entry=false) at osd/OSD.cc:6802 #7 0x000000000069fd56 in OSD::handle_pg_notify (this=0x4126000, op=...) at osd/OSD.cc:7310 #8 0x00000000006a2c58 in OSD::dispatch_op (this=this@entry=0x4126000, op=...) at osd/OSD.cc:5690 #9 0x00000000006a81d8 in OSD::_dispatch (this=this@entry=0x4126000, m=m@entry=0x333592a0) at osd/OSD.cc:5843 #10 0x00000000006a88a7 in OSD::ms_dispatch (this=0x4126000, m=0x333592a0) at osd/OSD.cc:5386 #11 0x0000000000c203e9 in ms_deliver_dispatch (m=0x333592a0, this=0x4068700) at msg/Messenger.h:532 #12 DispatchQueue::entry (this=0x40688b8) at msg/DispatchQueue.cc:185 #13 0x0000000000b5cd8d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #14 0x0000000005d72182 in start_thread (arg=0x19b29700) at pthread_create.c:312 #15 0x000000000784438d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
a bit more info:
(gdb) t 43
[Switching to thread 43 (Thread 21954)]
#0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33
33 ../nptl/sysdeps/x86_64/pthread_spin_lock.S: No such file or directory.
(gdb) bt
#0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:33
#1 0x000000000068c11a in ceph_spin_lock (l=0x41264a8) at ./include/Spinlock.h:45
#2 lock (this=0x41264a8) at ./include/Spinlock.h:94
#3 Locker (s=..., this=<synthetic pointer>) at ./include/Spinlock.h:105
#4 is_active (this=0x4126000) at osd/OSD.h:1074
#5 OSD::dispatch_context (this=this@entry=0x4126000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x21b38a70) at osd/OSD.cc:7129
#6 0x0000000000698977 in OSD::process_peering_events (this=0x4126000, pgs=..., handle=...) at osd/OSD.cc:8467
#7 0x00000000006edfb8 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1595
#8 0x0000000000b70ee6 in ThreadPool::worker (this=0x41264b0, wt=0xc412090) at common/WorkQueue.cc:128
#9 0x0000000000b71f90 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:318
#10 0x0000000005d72182 in start_thread (arg=0x21b39700) at pthread_create.c:312
#11 0x000000000784438d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) f 5
#5 OSD::dispatch_context (this=this@entry=0x4126000, ctx=..., pg=pg@entry=0x0, curmap=..., handle=handle@entry=0x21b38a70) at osd/OSD.cc:7129
7129 osd/OSD.cc: No such file or directory.
(gdb) do
#4 is_active (this=0x4126000) at osd/OSD.h:1074
1074 osd/OSD.h: No such file or directory.
(gdb) do
#3 Locker (s=..., this=<synthetic pointer>) at ./include/Spinlock.h:105
105 ./include/Spinlock.h: No such file or directory.
(gdb)
#2 lock (this=0x41264a8) at ./include/Spinlock.h:94
94 in ./include/Spinlock.h
(gdb) p state_lock
No symbol "state_lock" in current context.
(gdb) p _lock
$1 = {lock = -1}
(gdb) p &_lock
$2 = (ceph_spinlock_t *) 0x41264a8
(gdb) t 27
[Switching to thread 27 (Thread 21938)]
#0 pthread_spin_unlock () at ../nptl/sysdeps/x86_64/pthread_spin_unlock.S:23
23 ../nptl/sysdeps/x86_64/pthread_spin_unlock.S: No such file or directory.
(gdb) up
#1 0x00000000006597de in ceph_spin_unlock (l=0x41264a8) at ./include/Spinlock.h:50
50 ./include/Spinlock.h: No such file or directory.
(gdb) p _lock
No symbol "_lock" in current context.
(gdb) up
#2 unlock (this=0x41264a8) at ./include/Spinlock.h:98
98 in ./include/Spinlock.h
(gdb) p _lock
$3 = {lock = -1}
(gdb) p &_lock
$4 = (ceph_spinlock_t *) 0x41264a8
(same spinlock, value is -1)
Updated by Sage Weil over 9 years ago
Updated by Sage Weil over 9 years ago
valgrind is 1:3.10~20140411-0ubuntu1
3.10.0 release notes claim to have fixed
336435 Valgrind hangs in pthread_spin_lock consuming 100% CPU
opened a new valgrind bug, https://bugs.kde.org/show_bug.cgi?id=339537
Updated by Samuel Just over 9 years ago
- Status changed from Need More Info to Resolved
Actions