Actions
Bug #13385
closedcephx: verify_authorizer could not decrypt ticket info: error: NSS AES final round failed: -8190
Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Dan saw thing on bigbang, too.
2015-09-28 22:56:00.418474 7f3e46e34700 1 -- 10.214.130.125:6813/16283 >> :/0 pipe(0x7f3e7d4ac000 sd=28 :6813 s=0 pgs=0 cs=0 l=0 c=0x7f3e8de4b2e0).accept sd=28 10.214.130.53:45314/0 2015-09-28 22:56:00.419133 7f3e46e34700 0 cephx: verify_authorizer could not decrypt ticket info: error: NSS AES final round failed: -8190 2015-09-28 22:56:00.419165 7f3e46e34700 0 -- 10.214.130.125:6813/16283 >> 10.214.130.53:0/598934640 pipe(0x7f3e7d4ac000 sd=28 :6813 s=0 pgs=0 cs=0 l=1 c=0x7f3e8de4b2e0).accept: got bad authorizer
/a//teuthology-2015-09-27_17:10:01-upgrade:hammer-x-infernalis-distro-basic-vps/1072842
Updated by Sage Weil over 8 years ago
SEC_ERROR_BAD_DATA -8190 Security library: received bad data.
unsigned int written2;
ret = PK11_DigestFinal(ectx,
(unsigned char*)out_tmp.c_str()+written, &written2,
out_tmp.length()-written);
PK11_DestroyContext(ectx, PR_TRUE);
if (ret != SECSuccess) {
PK11_DestroyContext(ectx, PR_TRUE);
if (error) {
ostringstream oss;
oss << "NSS AES final round failed: " << PR_GetError();
*error = oss.str();
}
return -1;
}
(auth/Crypto.cc l 244)
Updated by Sage Weil over 8 years ago
Well, the rotating keyring used for authenticating is protected by a Mutex, but it hands out a reference, so it's pretty useless. I think we need ref counting on the CryptoKey's for this to be safe.
That said, I don't see that the rotating keys were getting updated when this happened.. there was no other auth activity around that time.
Updated by Sage Weil over 8 years ago
Thread 117 (Thread 0x7f3e48414700 (LWP 14551)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f3e70a15657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3e70a15480 in __GI___pthread_mutex_lock (mutex=0x7f3e77068fe8) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f3e7260f208 in Mutex::Lock (this=this@entry=0x7f3e77068fd8, no_lockdep=no_lockdep@entry=false) at common/Mutex.cc:102 #4 0x00007f3e727138d0 in Locker (m=..., this=<synthetic pointer>) at ./common/Mutex.h:115 #5 DispatchQueue::enqueue (this=0x7f3e77068fc8, m=0x7f3e799d7f80, priority=196, id=394) at msg/simple/DispatchQueue.cc:83 #6 0x00007f3e72737878 in Pipe::reader (this=0x7f3e8625b000) at msg/simple/Pipe.cc:1645 #7 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #8 0x00007f3e70a13182 in start_thread (arg=0x7f3e48414700) at pthread_create.c:312 #9 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 116 (Thread 0x7f3e4c73b700 (LWP 14548)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f3e70a1d428 in _L_cond_lock_886 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3e70a1d194 in __pthread_mutex_cond_lock (mutex=0x7f3e8625f160) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f3e70a174a4 in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:259 #4 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e8625f228) at ./common/Cond.h:55 #5 Pipe::writer (this=0x7f3e8625b000) at msg/simple/Pipe.cc:1832 #6 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #7 0x00007f3e70a13182 in start_thread (arg=0x7f3e4c73b700) at pthread_create.c:312 #8 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 115 (Thread 0x7f3e4783e700 (LWP 14481)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e4783d980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e87da5000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e87da5000, buf=buf@entry=0x7f3e4783db3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e87da5000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e4783e700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 114 (Thread 0x7f3e47036700 (LWP 14480)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7963f228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7963b000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e47036700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 113 (Thread 0x7f3e47339700 (LWP 14479)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e47338980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7963b000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7963b000, buf=buf@entry=0x7f3e47338b3f "\377\070", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7963b000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e47339700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 112 (Thread 0x7f3e47b12700 (LWP 14478)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e87da9228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e87da5000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e47b12700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 111 (Thread 0x7f3e4793f700 (LWP 14039)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e8ea2f228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e8ea2b000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4793f700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 110 (Thread 0x7f3e47238700 (LWP 14038)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e47237980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e79636000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e79636000, buf=buf@entry=0x7f3e47237b3f "\377\070", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e79636000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e47238700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 109 (Thread 0x7f3e46642700 (LWP 14037)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e46641980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e8ea2b000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e8ea2b000, buf=buf@entry=0x7f3e46641b3f "\377\002", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e8ea2b000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e46642700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ---Type <return> to continue, or q <return> to quit--- Thread 108 (Thread 0x7f3e71d5f700 (LWP 14036)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7963a228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e79636000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e71d5f700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 107 (Thread 0x7f3e46979700 (LWP 13484)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e8ea2a228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e8ea26000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e46979700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 106 (Thread 0x7f3e48515700 (LWP 13483)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48514980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e8ea26000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e8ea26000, buf=buf@entry=0x7f3e48514b3f "\377E", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e8ea26000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48515700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 105 (Thread 0x7f3e4743a700 (LWP 12836)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e47439980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e78a6f000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e78a6f000, buf=buf@entry=0x7f3e47439b3f "\377\002", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e78a6f000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e4743a700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 104 (Thread 0x7f3e4773d700 (LWP 12835)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e78a73228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e78a6f000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4773d700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 103 (Thread 0x7f3e46743700 (LWP 11646)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782dc228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782d8000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e46743700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 102 (Thread 0x7f3e4753b700 (LWP 11645)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e4753a980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782d8000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782d8000, buf=buf@entry=0x7f3e4753ab3f "\377\002", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782d8000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e4753b700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 101 (Thread 0x7f3e46f35700 (LWP 10668)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e46f34980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e87da0000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e87da0000, buf=buf@entry=0x7f3e46f34b3f "\377\307\001", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e87da0000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e46f35700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 100 (Thread 0x7f3e4ae37700 (LWP 10665)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7e6b5228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7e6b1000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4ae37700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 99 (Thread 0x7f3e4763c700 (LWP 10664)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e4763b980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7e6b1000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7e6b1000, buf=buf@entry=0x7f3e4763bb3f "\377\002", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7e6b1000) at msg/simple/Pipe.cc:1510 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e4763c700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 98 (Thread 0x7f3e46878700 (LWP 10661)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e87da4228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e87da0000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e46878700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 97 (Thread 0x7f3e48313700 (LWP 16934)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7131caf3 in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1 #2 0x00007f3e70a13182 in start_thread (arg=0x7f3e48313700) at pthread_create.c:312 #3 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 96 (Thread 0x7f3e48616700 (LWP 16522)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48615980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e781ab000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e781ab000, buf=buf@entry=0x7f3e48615b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e781ab000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48616700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 95 (Thread 0x7f3e48717700 (LWP 16521)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48716980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e781a6000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e781a6000, buf=buf@entry=0x7f3e48716b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e781a6000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48717700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 94 (Thread 0x7f3e48818700 (LWP 16520)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e781aa228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e781a6000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e48818700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 93 (Thread 0x7f3e48919700 (LWP 16519)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e781af228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e781ab000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e48919700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 92 (Thread 0x7f3e48a1a700 (LWP 16518)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e78450228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7844c000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e48a1a700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 91 (Thread 0x7f3e48b1b700 (LWP 16517)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782ce228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782ca000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e48b1b700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 90 (Thread 0x7f3e49020700 (LWP 16514)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e4901f980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782ca000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782ca000, buf=buf@entry=0x7f3e4901fb3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782ca000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49020700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 89 (Thread 0x7f3e48f1f700 (LWP 16513)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48f1e980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7844c000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7844c000, buf=buf@entry=0x7f3e48f1eb3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7844c000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 ---Type <return> to continue, or q <return> to quit--- #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48f1f700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 88 (Thread 0x7f3e4a333700 (LWP 16510)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7829e228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7829a000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4a333700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 87 (Thread 0x7f3e49727700 (LWP 16505)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49726980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7829a000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7829a000, buf=buf@entry=0x7f3e49726b3f "\377X\003", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7829a000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49727700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 86 (Thread 0x7f3e48c1c700 (LWP 16421)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48c1b980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e78456000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e78456000, buf=buf@entry=0x7f3e48c1bb3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e78456000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48c1c700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 85 (Thread 0x7f3e48d1d700 (LWP 16420)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e48d1c980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7845b000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7845b000, buf=buf@entry=0x7f3e48d1cb3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7845b000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e48d1d700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 84 (Thread 0x7f3e48e1e700 (LWP 16419)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7845f228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7845b000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e48e1e700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 83 (Thread 0x7f3e49222700 (LWP 16418)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e7845a228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e78456000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49222700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 82 (Thread 0x7f3e49424700 (LWP 16412)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e78455228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e78451000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49424700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 81 (Thread 0x7f3e49e2e700 (LWP 16411)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782c4228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782c0000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49e2e700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 80 (Thread 0x7f3e49a2a700 (LWP 16410)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49a29980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782cf000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782cf000, buf=buf@entry=0x7f3e49a29b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782cf000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49a2a700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 79 (Thread 0x7f3e49d2d700 (LWP 16406)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49d2c980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782c0000) at msg/simple/Pipe.cc:2400 ---Type <return> to continue, or q <return> to quit--- #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782c0000, buf=buf@entry=0x7f3e49d2cb3f "\377/", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782c0000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49d2d700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 78 (Thread 0x7f3e49929700 (LWP 16405)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49928980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e78451000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e78451000, buf=buf@entry=0x7f3e49928b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e78451000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49929700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 77 (Thread 0x7f3e49121700 (LWP 16403)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782d3228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782cf000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49121700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 76 (Thread 0x7f3e49323700 (LWP 16399)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782e1228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782dd000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49323700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 75 (Thread 0x7f3e49525700 (LWP 16398)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49524980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782dd000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782dd000, buf=buf@entry=0x7f3e49524b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782dd000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49525700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 74 (Thread 0x7f3e4a030700 (LWP 16396)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782e6228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782e2000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4a030700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 73 (Thread 0x7f3e49f2f700 (LWP 16395)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49f2e980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782e2000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782e2000, buf=buf@entry=0x7f3e49f2eb3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782e2000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49f2f700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 72 (Thread 0x7f3e49626700 (LWP 16380)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49625980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e7829f000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e7829f000, buf=buf@entry=0x7f3e49625b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e7829f000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49626700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 71 (Thread 0x7f3e49c2c700 (LWP 16377)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49c2b980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782c5000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782c5000, buf=buf@entry=0x7f3e49c2bb3f "\377\001", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782c5000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49c2c700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 70 (Thread 0x7f3e49828700 (LWP 16375)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72727707 in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7f3e49827980) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Pipe::tcp_read_wait (this=this@entry=0x7f3e782e7000) at msg/simple/Pipe.cc:2400 #3 0x00007f3e72727da8 in Pipe::tcp_read (this=this@entry=0x7f3e782e7000, buf=buf@entry=0x7f3e49827b3f "\377", len=len@entry=1) at msg/simple/Pipe.cc:2370 #4 0x00007f3e72735c8f in Pipe::reader (this=0x7f3e782e7000) at msg/simple/Pipe.cc:1510 #5 0x00007f3e7273f4cd in Pipe::Reader::entry (this=<optimized out>) at msg/simple/Pipe.h:50 ---Type <return> to continue, or q <return> to quit--- #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e49828700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 69 (Thread 0x7f3e49b2b700 (LWP 16368)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782eb228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782e7000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e49b2b700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 68 (Thread 0x7f3e4a131700 (LWP 16362)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782c9228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e782c5000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4a131700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 67 (Thread 0x7f3e4a232700 (LWP 16361)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7273c069 in Wait (mutex=..., this=0x7f3e782a3228) at ./common/Cond.h:55 #2 Pipe::writer (this=0x7f3e7829f000) at msg/simple/Pipe.cc:1832 #3 0x00007f3e7273f4ed in Pipe::Writer::entry (this=<optimized out>) at msg/simple/Pipe.h:62 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4a232700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 66 (Thread 0x7f3e4ad36700 (LWP 16351)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e72570251 in poll (__timeout=-1, __nfds=4, __fds=0x7f3e4ad35ba0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #2 SignalHandler::entry (this=0x7f3e7722f040) at global/signal_handler.cc:217 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e4ad36700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 65 (Thread 0x7f3e4b638700 (LWP 16347)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e720eeede in Cond::Wait (this=0x7f3e77229f78, mutex=...) at ./common/Cond.h:55 #2 0x00007f3e7208e50a in OSDService::agent_entry (this=0x7f3e77229758) at osd/OSD.cc:530 #3 0x00007f3e720f46fd in OSDService::AgentThread::entry (this=<optimized out>) at osd/OSD.h:662 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4b638700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 64 (Thread 0x7f3e4be39700 (LWP 16346)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7264636f in Wait (mutex=..., this=0x7f3e7722a0d0) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x7f3e7722a0c0) at common/Timer.cc:116 #3 0x00007f3e726478dd in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4be39700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 63 (Thread 0x7f3e4c63a700 (LWP 16345)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7264636f in Wait (mutex=..., this=0x7f3e7722a338) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x7f3e7722a328) at common/Timer.cc:116 #3 0x00007f3e726478dd in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e4c63a700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 62 (Thread 0x7f3e4cf3c700 (LWP 16343)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7259302c in Wait (mutex=..., this=0x7f3e7722a1f0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7f3e7722a190) at common/Finisher.cc:91 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e4cf3c700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 61 (Thread 0x7f3e4d73d700 (LWP 16342)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7259302c in Wait (mutex=..., this=0x7f3e7722a5d8) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7f3e7722a578) at common/Finisher.cc:91 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e4d73d700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 60 (Thread 0x7f3e4df3e700 (LWP 16341)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720b80dc in WaitUntil (when=..., mutex=..., this=0x7f3e77228ec0) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228ec0) at ./common/Cond.h:79 #3 OSD::heartbeat_entry (this=0x7f3e77228000) at osd/OSD.cc:3799 #4 0x00007f3e721198ed in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:1539 #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e4df3e700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 59 (Thread 0x7f3e4e73f700 (LWP 16340)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e7264d562 in WaitUntil (when=..., mutex=..., this=0x7f3e77228c60) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228c60) at common/Cond.h:79 #3 ThreadPool::worker (this=0x7f3e77228be8, wt=0x7f3e78034a20) at common/WorkQueue.cc:151 #4 0x00007f3e7264e7d0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e4e73f700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 58 (Thread 0x7f3e4ef40700 (LWP 16339)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e7264d562 in WaitUntil (when=..., mutex=..., this=0x7f3e77228ad0) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228ad0) at common/Cond.h:79 #3 ThreadPool::worker (this=0x7f3e77228a58, wt=0x7f3e78034a80) at common/WorkQueue.cc:151 #4 0x00007f3e7264e7d0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362 #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e4ef40700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 57 (Thread 0x7f3e4f741700 (LWP 16338)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e7264d562 in WaitUntil (when=..., mutex=..., this=0x7f3e77228940) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228940) at common/Cond.h:79 #3 ThreadPool::worker (this=0x7f3e772288c8, wt=0x7f3e78034b40) at common/WorkQueue.cc:151 #4 0x00007f3e7264e7d0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362 #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e4f741700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 56 (Thread 0x7f3e4ff42700 (LWP 16337)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e398) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e398) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036670) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e4ff42700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 55 (Thread 0x7f3e50743700 (LWP 16336)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e538) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e538) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036640) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e50743700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 54 (Thread 0x7f3e50f44700 (LWP 16335)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e6d8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e6d8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036610) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e50f44700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 53 (Thread 0x7f3e51745700 (LWP 16334)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e058) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e058) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e780365e0) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e51745700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 52 (Thread 0x7f3e51f46700 (LWP 16333)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e1f8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e1f8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e780365b0) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e51f46700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 51 (Thread 0x7f3e52747700 (LWP 16332)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e398) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e398) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036580) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e52747700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 50 (Thread 0x7f3e52f48700 (LWP 16331)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e538) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e538) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036550) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e52f48700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 49 (Thread 0x7f3e53749700 (LWP 16330)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e6d8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e6d8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e78036520) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e53749700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 48 (Thread 0x7f3e53f4a700 (LWP 16329)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e058) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e058) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e780364f0) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e53f4a700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 47 (Thread 0x7f3e5474b700 (LWP 16328)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e720e2489 in WaitUntil (when=..., mutex=..., this=0x7f3e7722e1f8) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e7722e1f8) at ./common/Cond.h:79 #3 OSD::ShardedOpWQ::_process (this=0x7f3e77229178, thread_index=<optimized out>, hb=0x7f3e780364c0) at osd/OSD.cc:8267 #4 0x00007f3e7264c8af in ShardedThreadPool::shardedthreadpool_worker (this=0x7f3e77228790, thread_index=<optimized out>) at common/WorkQueue.cc:340 #5 0x00007f3e7264e7b0 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at common/WorkQueue.h:592 #6 0x00007f3e70a13182 in start_thread (arg=0x7f3e5474b700) at pthread_create.c:312 #7 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 46 (Thread 0x7f3e54f4c700 (LWP 16327)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e7264d562 in WaitUntil (when=..., mutex=..., this=0x7f3e77228678) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228678) at common/Cond.h:79 #3 ThreadPool::worker (this=0x7f3e77228600, wt=0x7f3e773fe270) at common/WorkQueue.cc:151 #4 0x00007f3e7264e7d0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362 #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e54f4c700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 45 (Thread 0x7f3e5574d700 (LWP 16326)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e7264d562 in WaitUntil (when=..., mutex=..., this=0x7f3e77228678) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x7f3e77228678) at common/Cond.h:79 #3 ThreadPool::worker (this=0x7f3e77228600, wt=0x7f3e773fe210) at common/WorkQueue.cc:151 #4 0x00007f3e7264e7d0 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362 #5 0x00007f3e70a13182 in start_thread (arg=0x7f3e5574d700) at pthread_create.c:312 #6 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 44 (Thread 0x7f3e55f4e700 (LWP 16325)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e7259302c in Wait (mutex=..., this=0x7fff4cfecec0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7fff4cfece60) at common/Finisher.cc:91 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e55f4e700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 43 (Thread 0x7f3e5674f700 (LWP 16324)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f3e726465ef in WaitUntil (when=..., mutex=..., this=0x7fff4cfecda8) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x7fff4cfecd98) at common/Timer.cc:118 #3 0x00007f3e726478dd in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5674f700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 42 (Thread 0x7f3e56f50700 (LWP 16323)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e7706bbd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e7706b9c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e56f50700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 41 (Thread 0x7f3e57751700 (LWP 16322)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e7706ba30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e7706b9c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e57751700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (Thread 0x7f3e57f52700 (LWP 16321)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e726f44e9 in poll (__timeout=-1, __nfds=1, __fds=0x7f3e57f51ae0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7f3e77069d90) at msg/simple/Accepter.cc:227 ---Type <return> to continue, or q <return> to quit--- #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e57f52700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (Thread 0x7f3e58753700 (LWP 16320)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e77069fd0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e77069dc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e58753700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (Thread 0x7f3e58f54700 (LWP 16319)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e77069e30) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e77069dc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e58f54700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (Thread 0x7f3e59755700 (LWP 16318)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e726f44e9 in poll (__timeout=-1, __nfds=1, __fds=0x7f3e59754ae0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7f3e77069690) at msg/simple/Accepter.cc:227 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e59755700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (Thread 0x7f3e59f56700 (LWP 16317)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e770698d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e770696c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e59f56700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (Thread 0x7f3e5a757700 (LWP 16316)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e77069730) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e770696c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5a757700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (Thread 0x7f3e5af58700 (LWP 16315)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e7706a6d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e7706a4c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5af58700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (Thread 0x7f3e5b759700 (LWP 16314)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e7706a530) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e7706a4c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5b759700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (Thread 0x7f3e5bf5a700 (LWP 16313)): #0 0x00007f3e6ed4d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f3e726f44e9 in poll (__timeout=-1, __nfds=1, __fds=0x7f3e5bf59ae0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 Accepter::entry (this=0x7f3e77068890) at msg/simple/Accepter.cc:227 #3 0x00007f3e70a13182 in start_thread (arg=0x7f3e5bf5a700) at pthread_create.c:312 #4 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (Thread 0x7f3e5c75b700 (LWP 16312)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e77068ad0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e770688c8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5c75b700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (Thread 0x7f3e5cf5c700 (LWP 16311)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e77068930) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e770688c8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5cf5c700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (Thread 0x7f3e5d75d700 (LWP 16310)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f3e70a15657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f3e70a15480 in __GI___pthread_mutex_lock (mutex=0x7f3e77068fe8) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f3e7260f208 in Mutex::Lock (this=this@entry=0x7f3e77068fd8, no_lockdep=no_lockdep@entry=false) at common/Mutex.cc:102 #4 0x00007f3e7272bf0c in Locker (m=..., this=<synthetic pointer>) at ./common/Mutex.h:115 ---Type <return> to continue, or q <return> to quit--- #5 get_id (this=0x7f3e77068fc8) at msg/simple/DispatchQueue.h:185 #6 Pipe::Pipe (this=0x7f3e85885000, r=0x7f3e77068e00, st=0, con=0x0) at msg/simple/Pipe.cc:70 #7 0x00007f3e7263704d in SimpleMessenger::add_accept_pipe (this=0x7f3e77068e00, sd=sd@entry=29) at msg/simple/SimpleMessenger.cc:339 #8 0x00007f3e726f4623 in Accepter::entry (this=0x7f3e77068f90) at msg/simple/Accepter.cc:246 #9 0x00007f3e70a13182 in start_thread (arg=0x7f3e5d75d700) at pthread_create.c:312 #10 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (Thread 0x7f3e5df5e700 (LWP 16309)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e727134bc in Wait (mutex=..., this=0x7f3e770691d0) at ./common/Cond.h:55 #2 DispatchQueue::run_local_delivery (this=0x7f3e77068fc8) at msg/simple/DispatchQueue.cc:114 #3 0x00007f3e7263cb3d in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:117 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5df5e700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (Thread 0x7f3e5e75f700 (LWP 16308)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f3e72711d73 in Wait (mutex=..., this=0x7f3e77069030) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x7f3e77068fc8) at msg/simple/DispatchQueue.cc:196 #3 0x00007f3e7263cb1d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/simple/DispatchQueue.h:103 #4 0x00007f3e70a13182 in start_thread (arg=0x7f3e5e75f700) at pthread_create.c:312 #5 0x00007f3e6ed5a47d in clone () at ../sysdeps/unix/sysv/linux/x86
Updated by Kefu Chai over 8 years ago
Well, the rotating keyring used for authenticating is protected by a Mutex, but it hands out a reference, so it's pretty useless. I think we need ref counting on the CryptoKey's for this to be safe.
sage, i am not sure if i am following you. the indirect caller of nss_aes_operation() is cephx_verify_authorizer(). the later gets the CryptoKey from
- RotatingKeyRing::get_service_secret(uint32_t service_id_, uint64_t secret_id, CryptoKey& secret) or
- RotatingKeyRing::get_secret(const EntityName& name, CryptoKey& secret)
in both of these methods, we set the secret if we have it. and CryptoKey is holding a strong reference to ckh. so even the secrets in the rotating keyring are reset by CephxClientHandler after a CryptoKey is returned, we should always have a valid service_secret when decode_decrypt_enc_bl() gets called, right?
Updated by Kefu Chai over 8 years ago
- Assignee deleted (
Kefu Chai)
confirmed with sage that it's not the case that we are returning a reference to the caller. reassign myself from this ticket at this moment.
Updated by Sage Weil about 8 years ago
Hmm, we were seeing this a lot, but I haven't seen it recently.. has it gone away?
Updated by Ben England about 7 years ago
Just saw this bug , cluster was working normally one minute, and the next it's doing this on most of the OSDs. Never seen before. This is RHOSP 10, RHEL7.3 and RHCS 2.1?
heat-admin@overcloud-osd-compute-7:~
$ rpm -qa | grep ceph
ceph-base-10.2.5-37.el7cp.x86_64
python-cephfs-10.2.5-37.el7cp.x86_64
ceph-mon-10.2.5-37.el7cp.x86_64
ceph-radosgw-10.2.5-37.el7cp.x86_64
puppet-ceph-2.3.0-2.el7ost.noarch
libcephfs1-10.2.5-37.el7cp.x86_64
ceph-osd-10.2.5-37.el7cp.x86_64
ceph-common-10.2.5-37.el7cp.x86_64
ceph-selinux-10.2.5-37.el7cp.x86_64
$ more /etc/ceph/ceph.conf
[global]
osd_pool_default_pgp_num = 256
osd_pool_default_min_size = 2
auth_service_required = cephx
mon_initial_members = overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
fsid = eb2bb192-b1c9-11e6-9205-525400330666
cluster_network = 172.19.0.12/24
auth_supported = cephx
auth_cluster_required = cephx
mon_host = 172.18.0.15,172.18.0.10,172.18.0.17
auth_client_required = cephx
osd_pool_default_size = 3
osd_pool_default_pg_num = 256
public_network = 172.18.0.19/24
[osd]
osd_journal_size = 5120
Updated by Neha Ojha over 2 years ago
- Status changed from New to Can't reproduce
Actions