Project

General

Profile

Actions

Bug #8228

closed

0.80~rc1: OSD crash: segfault in libtcmalloc.so.4.1.2

Added by Dmitry Smirnov almost 10 years ago. Updated almost 10 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

OSD suddenly crashed on Debian GNU/Linux_3.14.1 x86_64; (~2 of ~12 GiB of RAM allocated).

From '/var/log/messages':

Apr 27 04:53:12 ceph-osd[19245]: segfault at 0 ip 00007ffb39292ce3 sp 00007ffb1f0204a0 error 4 in libtcmalloc.so.4.1.2[7ffb3926c000+46000]

Last entry in OSD log:

2014-04-27 04:52:57.350454 7ffb1f022700  0 log [INF] : 2.1a restarting backfill on osd.1 from (19228'122002,21317'125002] 0//0//-1 to 21461'125004
2014-04-27 04:53:00.437582 7ffb1e821700  0 log [INF] : 2.4 restarting backfill on osd.1 from (19879'222338,21417'225338] 0//0//-1 to 21465'225377

Backtrace from crash dump:

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff0c966000                                                                                          
Core was generated by `/usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.con'.                                                                                
Program terminated with signal 11, Segmentation fault.                                                                                                                                  
#0  0x00007ffb39292ce3 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /usr/lib/libtcmalloc.so.4                          
(gdb) thread apply all bt                                                                                                                                                               

Thread 147 (Thread 0x7ffb310e7700 (LWP 19223)):                                                                                                                                         
#0  0x00007ffb37578567 in ioctl () at ../sysdeps/unix/syscall-template.S:81                                                                                                             
#1  0x00007ffb39e650b5 in BtrfsFileStoreBackend::create_checkpoint (this=0x7ffb3bb215e0, name=..., transid=transid@entry=0x7ffb310e6aa0) at os/BtrfsFileStoreBackend.cc:381             
#2  0x00007ffb39d8adea in FileStore::sync_entry (this=0x7ffb3baf2000) at os/FileStore.cc:3132                                                                                           
#3  0x00007ffb39dc15dd in FileStore::SyncThread::entry (this=<optimized out>) at os/FileStore.h:178                                                                                     
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb310e7700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 146 (Thread 0x7ffb318e8700 (LWP 19222)):                                                                                                                                         
#0  0x00007ffb37578e6d in fdatasync () at ../sysdeps/unix/syscall-template.S:81                                                                                                         
#1  0x00007ffb39e24755 in WBThrottle::entry (this=0x7ffb3baf2998) at os/WBThrottle.cc:164                                                                                               
#2  0x00007ffb39057062 in start_thread (arg=0x7ffb318e8700) at pthread_create.c:312                                                                                                     
#3  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 145 (Thread 0x7ffb19211700 (LWP 19262)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb3dfa1af0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb3dfa1900) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb19211700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 144 (Thread 0x7ffb18605700 (LWP 19274)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18604b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb3dfa1180) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb3dfa1180, buf=buf@entry=0x7ffb18604cff "\377", len=len@entry=1) at msg/Pipe.cc:2245                                     
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb3dfa1180) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18605700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 143 (Thread 0x7ffb19110700 (LWP 19263)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb3dfa1870) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb3dfa1680) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb19110700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 142 (Thread 0x7ffb18d0c700 (LWP 19267)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb3dfa2270) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb3dfa2080) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb18d0c700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 141 (Thread 0x7ffb308e6700 (LWP 19224)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39b8f22e in Cond::Wait (this=0x7ffb3c1c6838, mutex=...) at ./common/Cond.h:55                                                                                             
#2  0x00007ffb39e87d90 in FileJournal::write_thread_entry (this=0x7ffb3c1c6700) at os/FileJournal.cc:1109                                                                               
#3  0x00007ffb39dbfe2d in FileJournal::Writer::entry (this=<optimized out>) at os/FileJournal.h:334                                                                                     
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb308e6700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 140 (Thread 0x7ffb286b9700 (LWP 20305)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb286b8b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1eaf80) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1eaf80, buf=buf@entry=0x7ffb286b8cff "\377/", len=len@entry=1) at msg/Pipe.cc:2245                                    
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1eaf80) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb286b9700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 139 (Thread 0x7ffb2bc05700 (LWP 19229)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39ec7798 in Wait (mutex=..., this=0x7ffb3baf2cc0) at ./common/Cond.h:55                                                                                                   
#2  Finisher::finisher_thread_entry (this=0x7ffb3baf2c60) at common/Finisher.cc:78                                                                                                      
#3  0x00007ffb39057062 in start_thread (arg=0x7ffb2bc05700) at pthread_create.c:312                                                                                                     
#4  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 138 (Thread 0x7ffb3569b700 (LWP 18926)):                                                                                                                                         
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238                                                                       
#1  0x00007ffb39f9aa03 in WaitUntil (when=..., mutex=..., this=0x7ffb3baf8428) at ./common/Cond.h:71                                                                                    
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7ffb3baf8428) at ./common/Cond.h:79                                                                              
#3  CephContextServiceThread::entry (this=0x7ffb3baf83c0) at common/ceph_context.cc:58                                                                                                  
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb3569b700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 137 (Thread 0x7ffb36618700 (LWP 18923)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39ed852b in ceph::log::Log::entry (this=0x7ffb3badc000) at log/Log.cc:323                                                                                                 
#2  0x00007ffb39057062 in start_thread (arg=0x7ffb36618700) at pthread_create.c:312                                                                                                     
#3  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 136 (Thread 0x7ffb2502e700 (LWP 19233)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a011459 in poll (__timeout=-1, __nfds=1, __fds=0x7ffb2502dca0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                                          
#2  Accepter::entry (this=0x7ffb3bb0c0c0) at msg/Accepter.cc:196                                                                                                                        
#3  0x00007ffb39057062 in start_thread (arg=0x7ffb2502e700) at pthread_create.c:312                                                                                                     
#4  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 135 (Thread 0x7ffb398a77c0 (LWP 18922)):                                                                                                                                         
#0  0x00007ffb3905848b in pthread_join (threadid=140716642858752, thread_return=0x0) at pthread_join.c:92                                                                               

#1  0x00007ffb39f6de82 in Thread::join (this=this@entry=0x7ffb3bb0c270, prval=prval@entry=0x0) at common/Thread.cc:121                                                                  
#2  0x00007ffb3a0230ce in DispatchQueue::wait (this=this@entry=0x7ffb3bb0c0e8) at msg/DispatchQueue.cc:167                                                                              
#3  0x00007ffb39f64ab9 in SimpleMessenger::wait (this=0x7ffb3bb0c000) at msg/SimpleMessenger.cc:527                                                                                     
#4  0x00007ffb39b23a98 in main (argc=<optimized out>, argv=<optimized out>) at ceph_osd.cc:502                                                                                          

Thread 134 (Thread 0x7ffb2402c700 (LWP 19235)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a011459 in poll (__timeout=-1, __nfds=1, __fds=0x7ffb2402bca0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                                          
#2  Accepter::entry (this=0x7ffb3bb0ccc0) at msg/Accepter.cc:196                                                                                                #3  0x00007ffb39057062 in start_thread (arg=0x7ffb2402c700) at pthread_create.c:312                                                                                                     
#4  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 133 (Thread 0x7ffb15fe6700 (LWP 22701)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb15fe5b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1e9b80) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1e9b80, buf=buf@entry=0x7ffb15fe5cff "\377/", len=len@entry=1) at msg/Pipe.cc:2245                                    
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1e9b80) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb15fe6700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 132 (Thread 0x7ffb21827700 (LWP 19240)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a011459 in poll (__timeout=-1, __nfds=1, __fds=0x7ffb21826ca0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                                          
#2  Accepter::entry (this=0x7ffb3bb0dec0) at msg/Accepter.cc:196                                                                                                                        
#3  0x00007ffb39057062 in start_thread (arg=0x7ffb21827700) at pthread_create.c:312                                                                                                     
#4  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 131 (Thread 0x7ffb13dc4700 (LWP 22696)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb13dc3b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb3dfa0780) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb3dfa0780, buf=buf@entry=0x7ffb13dc3cff "\377\230?SR\373\177", len=len@entry=1) at msg/Pipe.cc:2245                      
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb3dfa0780) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb13dc4700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 130 (Thread 0x7ffb19514700 (LWP 22683)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb19513b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e854000) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e854000, buf=buf@entry=0x7ffb19513cff "\377\350LPZ\373\177", len=len@entry=1) at msg/Pipe.cc:2245                      
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e854000) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb19514700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 129 (Thread 0x7ffb13cc3700 (LWP 22680)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb13cc2b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5d0ae280) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5d0ae280, buf=buf@entry=0x7ffb13cc2cff "\377\330\017\224^\373\177", len=len@entry=1) at msg/Pipe.cc:2245                
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5d0ae280) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb13cc3700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 128 (Thread 0x7ffb154db700 (LWP 22211)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5bf2b870) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5bf2b680) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb154db700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 127 (Thread 0x7ffb1900f700 (LWP 22210)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb1900eb50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5bf2b680) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5bf2b680, buf=buf@entry=0x7ffb1900ecff "\377\002", len=len@entry=1) at msg/Pipe.cc:2245                                 
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5bf2b680) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb1900f700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 126 (Thread 0x7ffb18706700 (LWP 20311)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18705b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1e8a00) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1e8a00, buf=buf@entry=0x7ffb18705cff "\377\036", len=len@entry=1) at msg/Pipe.cc:2245                                 
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1e8a00) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18706700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 125 (Thread 0x7ffb2302a700 (LWP 19237)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a023c3b in Wait (mutex=..., this=0x7ffb3bb0d950) at ./common/Cond.h:55                                                                                                   
#2  DispatchQueue::entry (this=0x7ffb3bb0d8e8) at msg/DispatchQueue.cc:138                                                                                                              
#3  0x00007ffb39f69cad in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104                                                                        
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb2302a700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 124 (Thread 0x7ffb12d62700 (LWP 19363)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5de58bf0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5de58a00) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb12d62700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 123 (Thread 0x7ffb18403700 (LWP 20310)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18402b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1e8c80) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1e8c80, buf=buf@entry=0x7ffb18402cff "\377\230\031\376^\373\177", len=len@entry=1) at msg/Pipe.cc:2245                
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1e8c80) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18403700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 122 (Thread 0x7ffb165ec700 (LWP 19311)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5b5935f0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5b593400) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb165ec700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 121 (Thread 0x7ffb14ad1700 (LWP 20306)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5e1eb170) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5e1eaf80) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb14ad1700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 120 (Thread 0x7ffb1a819700 (LWP 19254)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb3dfa0470) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb3dfa0280) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb1a819700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 119 (Thread 0x7ffb16ef5700 (LWP 20296)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5e1e90f0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5e1e8f00) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb16ef5700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 118 (Thread 0x7ffb14cd3700 (LWP 20297)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5e1ea270) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5e1ea080) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb14cd3700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 117 (Thread 0x7ffb2482d700 (LWP 19234)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a023c3b in Wait (mutex=..., this=0x7ffb3bb0cd50) at ./common/Cond.h:55                                                                                                   
#2  DispatchQueue::entry (this=0x7ffb3bb0cce8) at msg/DispatchQueue.cc:138                                                                                                              
#3  0x00007ffb39f69cad in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104                                                                        
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb2482d700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 116 (Thread 0x7ffb18807700 (LWP 20293)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18806b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1e8f00) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1e8f00, buf=buf@entry=0x7ffb18806cff "\377\002", len=len@entry=1) at msg/Pipe.cc:2245                                 
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1e8f00) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18807700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 115 (Thread 0x7ffb18a09700 (LWP 19270)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb3dfa1370) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb3dfa1180) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb18a09700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 114 (Thread 0x7ffb2e40a700 (LWP 19227)):                                                                                                                                         
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238                                                                       
#1  0x00007ffb39f7776c in WaitUntil (when=..., mutex=..., this=0x7ffb3baf2e08) at common/Cond.h:71                                                                                      
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7ffb3baf2e08) at common/Cond.h:79                                                                                
#3  ThreadPool::worker (this=0x7ffb3baf2d90, wt=0x7ffb3bb23f00) at common/WorkQueue.cc:143                                                                                              
#4  0x00007ffb39f78aa0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317                                                                                
#5  0x00007ffb39057062 in start_thread (arg=0x7ffb2e40a700) at pthread_create.c:312                                                                                                     
#6  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 113 (Thread 0x7ffb19f17700 (LWP 19256)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39b8f22e in Cond::Wait (this=0x7ffb3c0e2d28, mutex=...) at ./common/Cond.h:55                                                                                             
#2  0x00007ffb39b3292c in OSDService::agent_entry (this=0x7ffb3c0e26b8) at osd/OSD.cc:493                                                                                               
#3  0x00007ffb39b971ad in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:464                                                                                        
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb19f17700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 112 (Thread 0x7ffb18e0d700 (LWP 19266)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18e0cb50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb3dfa1900) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb3dfa1900, buf=buf@entry=0x7ffb18e0ccff "\377", len=len@entry=1) at msg/Pipe.cc:2245                                     
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb3dfa1900) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18e0d700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 111 (Thread 0x7ffb2ec0b700 (LWP 19226)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39ec7798 in Wait (mutex=..., this=0x7ffb3baf2080) at ./common/Cond.h:55                                                                                                   
#2  Finisher::finisher_thread_entry (this=0x7ffb3baf2020) at common/Finisher.cc:78                                                                                                      
#3  0x00007ffb39057062 in start_thread (arg=0x7ffb2ec0b700) at pthread_create.c:312                                                                                                     
#4  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 110 (Thread 0x7ffb19716700 (LWP 19257)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb19715b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb3dfa0280) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb3dfa0280, buf=buf@entry=0x7ffb19715cff "\377", len=len@entry=1) at msg/Pipe.cc:2245                                     
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb3dfa0280) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb19716700) at pthread_create.c:312                                                                                               #7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 109 (Thread 0x7ffb13166700 (LWP 20292)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5bf2b5f0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5bf2b400) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb13166700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 108 (Thread 0x7ffb18908700 (LWP 19271)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb18907b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb3dfa2080) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb3dfa2080, buf=buf@entry=0x7ffb18907cff "\377", len=len@entry=1) at msg/Pipe.cc:2245                                     
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb3dfa2080) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb18908700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 107 (Thread 0x7ffb2f40c700 (LWP 18950)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39f63d3f in Wait (mutex=..., this=0x7ffb3bb0e318) at ./common/Cond.h:55                                                                                                   
#2  SimpleMessenger::reaper_entry (this=0x7ffb3bb0de00) at msg/SimpleMessenger.cc:207                                                                                                   
#3  0x00007ffb39f6a67d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225                                                                      
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb2f40c700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 106 (Thread 0x7ffb160e7700 (LWP 19316)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5b5944f0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5b594300) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb160e7700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 105 (Thread 0x7ffb31a7f700 (LWP 19571)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb31a7eb50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5de58780) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5de58780, buf=buf@entry=0x7ffb31a7ecff "\377", len=len@entry=1) at msg/Pipe.cc:2245                                     
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5de58780) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb31a7f700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 104 (Thread 0x7ffb2cc07700 (LWP 18948)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb39f63d3f in Wait (mutex=..., this=0x7ffb3bb0cb18) at ./common/Cond.h:55                                                                                                   
#2  SimpleMessenger::reaper_entry (this=0x7ffb3bb0c600) at msg/SimpleMessenger.cc:207                                                                                                   
#3  0x00007ffb39f6a67d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:225                                                                      
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb2cc07700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     
Thread 103 (Thread 0x7ffb13065700 (LWP 20304)):                                                                                                                                         
#0  0x00007ffb3757472d in poll () at ../sysdeps/unix/syscall-template.S:81                                                                                                              
#1  0x00007ffb3a037a0f in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7ffb13064b50) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46                                             
#2  Pipe::tcp_read_wait (this=this@entry=0x7ffb5e1e8000) at msg/Pipe.cc:2272                                                                                                            
#3  0x00007ffb3a037cf8 in Pipe::tcp_read (this=this@entry=0x7ffb5e1e8000, buf=buf@entry=0x7ffb13064cff "\377\350hQU\373\177", len=len@entry=1) at msg/Pipe.cc:2245                      
#4  0x00007ffb3a04ae4f in Pipe::reader (this=0x7ffb5e1e8000) at msg/Pipe.cc:1411                                                                                                        
#5  0x00007ffb3a04f3fd in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:49                                                                                                   
#6  0x00007ffb39057062 in start_thread (arg=0x7ffb13065700) at pthread_create.c:312                                                                                                     
#7  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 102 (Thread 0x7ffb16ff6700 (LWP 20298)):                                                                                                                                         
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185                                                                                 
#1  0x00007ffb3a043a58 in Wait (mutex=..., this=0x7ffb5e1e86f0) at ./common/Cond.h:55                                                                                                   
#2  Pipe::writer (this=0x7ffb5e1e8500) at msg/Pipe.cc:1720                                                                                                                              
#3  0x00007ffb3a04ec9d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:61                                                                                                   
#4  0x00007ffb39057062 in start_thread (arg=0x7ffb16ff6700) at pthread_create.c:312                                                                                                     
#5  0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111                                                                                                     

Thread 101 (Thread 0x7ffb12e63700 (LWP 19361)):                                                    

Actions #1

Updated by Dmitry Smirnov almost 10 years ago

(gdb) where
#0  0x00007ffb39292ce3 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /usr/lib/libtcmalloc.so.4
#1  0x00007ffb39292d9b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /usr/lib/libtcmalloc.so.4
#2  0x00007ffb392a0648 in tc_free () from /usr/lib/libtcmalloc.so.4
#3  0x00007ffb37dc4a29 in __cxa_demangle () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffb39ea808b in ceph::BackTrace::print (this=this@entry=0x7ffb1f0208b0, out=...) at common/BackTrace.cc:47
#5  0x00007ffb39ea5de4 in handle_fatal_signal (signum=11) at global/signal_handler.cc:92
#6  <signal handler called>
#7  0x00007ffb39292ce3 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /usr/lib/libtcmalloc.so.4
#8  0x00007ffb39292d9b in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) () from /usr/lib/libtcmalloc.so.4
#9  0x00007ffb392a2188 in tc_delete () from /usr/lib/libtcmalloc.so.4
#10 0x00007ffb39b8a861 in operator() (this=<optimized out>, __p=0x7ffb5edd6b20) at /usr/include/c++/4.8/tr1/shared_ptr.h:285
#11 std::tr1::_Sp_counted_base_impl<PG::CephPeeringEvt*, std::tr1::_Sp_deleter<PG::CephPeeringEvt>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=<optimized out>)
    at /usr/include/c++/4.8/tr1/shared_ptr.h:257
#12 0x00007ffb39b96b29 in std::tr1::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7ffb5edd7500) at /usr/include/c++/4.8/tr1/shared_ptr.h:141
#13 0x00007ffb39b7a7fb in ~__shared_count (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:341
#14 ~__shared_ptr (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:541
#15 ~shared_ptr (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/c++/4.8/tr1/shared_ptr.h:985
#16 OSD::process_peering_events (this=0x7ffb3c0e1000, pgs=..., handle=...) at osd/OSD.cc:7722
#17 0x00007ffb39bc1a78 in OSD::PeeringWQ::_process (this=<optimized out>, pgs=..., handle=...) at osd/OSD.h:1199
#18 0x00007ffb39f77bb1 in ThreadPool::worker (this=0x7ffb3c0e1470, wt=0x7ffb4165afa0) at common/WorkQueue.cc:125
#19 0x00007ffb39f78aa0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:317
#20 0x00007ffb39057062 in start_thread (arg=0x7ffb1f022700) at pthread_create.c:312
#21 0x00007ffb3757fa3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Actions #2

Updated by Ian Colle almost 10 years ago

  • Assignee set to Samuel Just
  • Priority changed from Normal to Urgent
Actions #3

Updated by Samuel Just almost 10 years ago

Is there a straightforward way to reproduce this?

Actions #4

Updated by Dmitry Smirnov almost 10 years ago

I only had it once, it never happened again as far as I'm aware...
No ideas what could be the cause.
I suspect that RAM defect could be involved but that's just guessing...

Actions #5

Updated by Samuel Just almost 10 years ago

  • Status changed from New to Can't reproduce

In that case, marking can't reproduce. Let us know if it recurs.

Actions

Also available in: Atom PDF