Project

General

Profile

Bug #20748

segfault in OSDMap::_pg_to_up_acting_osds

Added by Mark Nelson over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
OSDMap
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

log: incerta03:/tmp/cbt/ceph/log/osd.10.log
core: incerta03:/tmp/cbt/ceph/core.tp_osd_tp.17701.incerta03.front.sepia.ceph.com.1500771680
bin: incerta03:/usr/local/bin/ceph-osd

bt:

Thread 65 (Thread 0x7f1b2c44d700 (LWP 17702)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded95931b in ceph::logging::Log::entry (this=0x560df7b88000) at /home/ubuntu/src/markhpc/ceph/src/log/Log.cc:454
#2  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b2c44d700) at pthread_create.c:308
#3  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 64 (Thread 0x7f1b09c5c700 (LWP 17785)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded4212be in Cond::Wait (this=0x560df7d6ff90, mutex=...) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  0x0000560ded3b1740 in OSDService::agent_entry (this=0x560df7d6f860) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:621
#3  0x0000560ded428d9d in OSDService::AgentThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.h:665
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b09c5c700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 63 (Thread 0x7f1b0ec66700 (LWP 17775)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04e18) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04e18) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b600) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0ec66700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 62 (Thread 0x7f1b0f467700 (LWP 17774)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04cd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04cd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b5d0) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0f467700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 61 (Thread 0x7f1b0fc68700 (LWP 17773)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04b98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04b98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b5a0) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0fc68700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 60 (Thread 0x7f1b10469700 (LWP 17772)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04a58) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04a58) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b570) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b10469700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 59 (Thread 0x7f1b10c6a700 (LWP 17771)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b540) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b10c6a700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 58 (Thread 0x7f1b1146b700 (LWP 17770)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a047d8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
---Type <return> to continue, or q <return> to quit---
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a047d8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b510) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1146b700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 57 (Thread 0x7f1b11c6c700 (LWP 17769)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04698) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04698) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b4e0) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b11c6c700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 56 (Thread 0x7f1b1246d700 (LWP 17768)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04558) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04558) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b4b0) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1246d700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 55 (Thread 0x7f1b12c6e700 (LWP 17767)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04e18) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04e18) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b480) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b12c6e700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 54 (Thread 0x7f1b1346f700 (LWP 17766)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04cd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04cd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b450) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1346f700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 53 (Thread 0x7f1b13c70700 (LWP 17765)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04b98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04b98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b420) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b13c70700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 52 (Thread 0x7f1b0a45d700 (LWP 17784)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d70c38) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d70c28) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0a45d700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 51 (Thread 0x7f1b14471700 (LWP 17764)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04a58) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
---Type <return> to continue, or q <return> to quit---
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04a58) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b3f0) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b14471700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 50 (Thread 0x7f1b0ac5e700 (LWP 17783)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d70b18) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d70b08) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0ac5e700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 49 (Thread 0x7f1b15c74700 (LWP 17761)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04698) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04698) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b360) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b15c74700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 48 (Thread 0x7f1b0b45f700 (LWP 17782)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d700f0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d700e0) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0b45f700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 47 (Thread 0x7f1b17477700 (LWP 17758)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded950baa in WaitUntil (when=..., mutex=..., this=0x560df7d6e918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7d6e918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  ThreadPool::worker (this=0x560df7d6e898, wt=0x560df8989c80) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:145
#4  0x0000560ded951f70 in ThreadPool::WorkThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:448
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b17477700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 46 (Thread 0x7f1b0bc60700 (LWP 17781)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d703a0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d70390) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0bc60700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 45 (Thread 0x7f1b1a47d700 (LWP 17752)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bb9f70) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bb9d58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1a47d700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 44 (Thread 0x7f1b1ac7e700 (LWP 17751)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bb9dc0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bb9d58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1ac7e700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 43 (Thread 0x7f1b16c76700 (LWP 17759)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
---Type <return> to continue, or q <return> to quit---
#1  0x0000560ded950baa in WaitUntil (when=..., mutex=..., this=0x560df7d6e918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7d6e918) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  ThreadPool::worker (this=0x560df7d6e898, wt=0x560df8989c20) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:145
#4  0x0000560ded951f70 in ThreadPool::WorkThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:448
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b16c76700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 42 (Thread 0x7f1b0c461700 (LWP 17780)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9481c1 in Wait (mutex=..., this=0x560df7d70248) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  Finisher::finisher_thread_entry (this=0x560df7d701e8) at /home/ubuntu/src/markhpc/ceph/src/common/Finisher.cc:101
#3  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0c461700) at pthread_create.c:308
#4  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 41 (Thread 0x7f1b1b47f700 (LWP 17750)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bba670) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bba458) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1b47f700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 40 (Thread 0x7f1b17c78700 (LWP 17757)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d6e3f0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d6e3e0) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b17c78700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 39 (Thread 0x7f1b0cc62700 (LWP 17779)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9481c1 in Wait (mutex=..., this=0x560df7d70718) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  Finisher::finisher_thread_entry (this=0x560df7d706b8) at /home/ubuntu/src/markhpc/ceph/src/common/Finisher.cc:101
#3  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0cc62700) at pthread_create.c:308
#4  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 38 (Thread 0x7f1b1e524700 (LWP 17748)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bb9170) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bb8f58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1e524700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 37 (Thread 0x7f1b18479700 (LWP 17756)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9481c1 in Wait (mutex=..., this=0x7ffc6df403d0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  Finisher::finisher_thread_entry (this=0x7ffc6df40370) at /home/ubuntu/src/markhpc/ceph/src/common/Finisher.cc:101
#3  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b18479700) at pthread_create.c:308
#4  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 36 (Thread 0x7f1b0d463700 (LWP 17778)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded3d32d1 in WaitUntil (when=..., mutex=..., this=0x560df7d6ef98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7d6ef98) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::heartbeat_entry (this=0x560df7d6e000) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:4846
#4  0x0000560ded4448cd in OSD::T_Heartbeat::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.h:1527
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0d463700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 35 (Thread 0x7f1b1f526700 (LWP 17746)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bb9870) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bb9658) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1f526700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
---Type <return> to continue, or q <return> to quit---

Thread 34 (Thread 0x7f1b1947b700 (LWP 17754)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bbad70) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bbab58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1947b700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 33 (Thread 0x7f1b19c7c700 (LWP 17753)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bbabc0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bbab58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b19c7c700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 32 (Thread 0x7f1b20528700 (LWP 17744)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bb8a70) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bb8858) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b20528700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 31 (Thread 0x7f1b1bc80700 (LWP 17749)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bba4c0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bba458) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1bc80700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 30 (Thread 0x7f1b0dc64700 (LWP 17777)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded950baa in WaitUntil (when=..., mutex=..., this=0x560df7d6ed70) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7d6ed70) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  ThreadPool::worker (this=0x560df7d6ecf0, wt=0x560df8989d70) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:145
#4  0x0000560ded951f70 in ThreadPool::WorkThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:448
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0dc64700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 29 (Thread 0x7f1b20d29700 (LWP 17743)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bb88c0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bb8858) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b20d29700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 28 (Thread 0x7f1b16475700 (LWP 17760)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a04558) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a04558) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b330) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b16475700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 27 (Thread 0x7f1b1d483700 (LWP 17738)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f1b2da289ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x0)
    at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/x86_64-redhat-linux/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=this@entry=0x560df7ba6640, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:52
#3  0x0000560ded7ea104 in BlueStore::_kv_finalize_thread (this=0x560df7ba6000) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/BlueStore.cc:8508
#4  0x0000560ded83e34d in BlueStore::KVFinalizeThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/BlueStore.h:1781
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1d483700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 26 (Thread 0x7f1b1dc84700 (LWP 17739)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded7b8ec4 in WaitUntil (when=..., mutex=..., this=0x560df7ba6928) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7ba6928) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  BlueStore::MempoolThread::entry (this=0x560df7ba68f8) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/BlueStore.cc:3345
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1dc84700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 25 (Thread 0x7f1b1ed25700 (LWP 17747)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bb8fc0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bb8f58) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1ed25700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 24 (Thread 0x7f1b15473700 (LWP 17762)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded408dda in WaitUntil (when=..., mutex=..., this=0x560df7a047d8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7a047d8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=0x560df898b390) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:9961
#4  0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#5  0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#6  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b15473700) at pthread_create.c:308
#7  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 23 (Thread 0x7f1b0e465700 (LWP 17776)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded950baa in WaitUntil (when=..., mutex=..., this=0x560df7d6ebd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7d6ebd8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  ThreadPool::worker (this=0x560df7d6eb58, wt=0x560df8989d10) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:145
#4  0x0000560ded951f70 in ThreadPool::WorkThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:448
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b0e465700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 22 (Thread 0x7f1b2152a700 (LWP 17742)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb4651f in Wait (mutex=..., this=0x560df7bb8370) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::run_local_delivery (this=0x560df7bb8158) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:113
#3  0x0000560ded9d39bd in DispatchQueue::LocalDeliveryThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:115
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b2152a700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 21 (Thread 0x7f1b24c92700 (LWP 17716)):
#0  0x00007f1b301ca644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000560ded9059c7 in aio_queue_t::get_next_completed (this=this@entry=0x560df7c80d48, timeout_ms=<optimized out>, paio=paio@entry=0x7f1b24c90870, max=16) at /home/ubuntu/src/markhpc/ceph/src/os/fs/aio.cc:74
#2  0x0000560ded8f0327 in KernelDevice::_aio_thread (this=0x560df7c80c00) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.cc:345
#3  0x0000560ded8f4dad in KernelDevice::AioCompletionThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.h:49
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b24c92700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 20 (Thread 0x7f1b18c7a700 (LWP 17755)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded945b9e in WaitUntil (when=..., mutex=..., this=0x7ffc6df402b8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  SafeTimer::timer_thread (this=0x7ffc6df402a8) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:110
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b18c7a700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 19 (Thread 0x7f1b1fd27700 (LWP 17745)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bb96c0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bb9658) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1fd27700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
---Type <return> to continue, or q <return> to quit---

Thread 18 (Thread 0x7f1b1cc82700 (LWP 17737)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f1b2da289ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x6f7ddba00)
    at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/x86_64-redhat-linux/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=this@entry=0x560df7ba6420, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:52
#3  0x0000560ded7fe3cc in BlueStore::_kv_sync_thread (this=0x560df7ba6000) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/BlueStore.cc:8262
#4  0x0000560ded843c1d in BlueStore::KVSyncThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/BlueStore.h:1773
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1cc82700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 17 (Thread 0x7f1b24491700 (LWP 17717)):
#0  0x00007f1b301ca644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000560ded9059c7 in aio_queue_t::get_next_completed (this=this@entry=0x560df7c80348, timeout_ms=<optimized out>, paio=paio@entry=0x7f1b2448f870, max=16) at /home/ubuntu/src/markhpc/ceph/src/os/fs/aio.cc:74
#2  0x0000560ded8f0327 in KernelDevice::_aio_thread (this=0x560df7c80200) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.cc:345
#3  0x0000560ded8f4dad in KernelDevice::AioCompletionThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.h:49
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b24491700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 16 (Thread 0x7f1b1c481700 (LWP 17736)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9481c1 in Wait (mutex=..., this=0x560df7af0a00) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  Finisher::finisher_thread_entry (this=0x560df7af09a0) at /home/ubuntu/src/markhpc/ceph/src/common/Finisher.cc:101
#3  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b1c481700) at pthread_create.c:308
#4  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 15 (Thread 0x7f1b25493700 (LWP 17715)):
#0  0x00007f1b301ca644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000560ded9059c7 in aio_queue_t::get_next_completed (this=this@entry=0x560df7c81948, timeout_ms=<optimized out>, paio=paio@entry=0x7f1b25491870, max=16) at /home/ubuntu/src/markhpc/ceph/src/os/fs/aio.cc:74
#2  0x0000560ded8f0327 in KernelDevice::_aio_thread (this=0x560df7c81800) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.cc:345
#3  0x0000560ded8f4dad in KernelDevice::AioCompletionThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.h:49
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b25493700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 14 (Thread 0x7f1b25c94700 (LWP 17714)):
#0  0x00007f1b301ca644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1  0x0000560ded9059c7 in aio_queue_t::get_next_completed (this=this@entry=0x560df7c81748, timeout_ms=<optimized out>, paio=paio@entry=0x7f1b25c92870, max=16) at /home/ubuntu/src/markhpc/ceph/src/os/fs/aio.cc:74
#2  0x0000560ded8f0327 in KernelDevice::_aio_thread (this=0x560df7c81600) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.cc:345
#3  0x0000560ded8f4dad in KernelDevice::AioCompletionThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/os/bluestore/KernelDevice.h:49
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b25c94700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 13 (Thread 0x7f1b26495700 (LWP 17713)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d705f8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d705e8) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b26495700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 12 (Thread 0x7f1b27497700 (LWP 17711)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded945b9e in WaitUntil (when=..., mutex=..., this=0x560df7d6e1a0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  SafeTimer::timer_thread (this=0x560df7d6e190) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:110
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b27497700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 11 (Thread 0x7f1b21d2b700 (LWP 17741)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560dedb439fb in Wait (mutex=..., this=0x560df7bb81c0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  DispatchQueue::entry (this=0x560df7bb8158) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.cc:208
#3  0x0000560ded9d399d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/DispatchQueue.h:101
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b21d2b700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 10 (Thread 0x7f1b2bc4c700 (LWP 17705)):
#0  0x00007f1b2d195d13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
---Type <return> to continue, or q <return> to quit---
#1  0x0000560dedaf8e64 in EpollDriver::event_wait (this=0x560df7ba0e20, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/async/EventEpoll.cc:114
#2  0x0000560ded9e3cba in EventCenter::process_events (this=this@entry=0x560df7a1a680, timeout_microseconds=<optimized out>, timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7f1b2bc4a930)
    at /home/ubuntu/src/markhpc/ceph/src/msg/async/Event.cc:395
#3  0x0000560ded9e6abe in NetworkStack::__lambda4::operator() (__closure=0x560df79d84f0) at /home/ubuntu/src/markhpc/ceph/src/msg/async/Stack.cc:51
#4  0x00007f1b2da2c230 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b2bc4c700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 9 (Thread 0x7f1b29cd7700 (LWP 17706)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560deda3cf1a in WaitUntil (when=..., mutex=..., this=0x560df7b80080) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  WaitInterval (interval=..., mutex=..., this=0x560df7b80080) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:73
#3  CephContextServiceThread::entry (this=0x560df7b80000) at /home/ubuntu/src/markhpc/ceph/src/common/ceph_context.cc:135
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b29cd7700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 8 (Thread 0x7f1b30a44b80 (LWP 17701)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded42151c in Wait (mutex=..., this=0x7ffc6df3ecf0) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  C_SaferCond::wait (this=0x7ffc6df3ec90) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:194
#3  0x0000560ded3bdb46 in OSD::mon_cmd_maybe_osd_create (this=this@entry=0x560df7d6e000, cmd="{\"prefix\": \"osd crush set-device-class\", \"class\": \"nvme\", \"ids\": [\"10\"]}")
    at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:3402
#4  0x0000560ded3be59a in OSD::update_crush_device_class (this=this@entry=0x560df7d6e000) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:3493
#5  0x0000560ded406c0e in OSD::init (this=0x560df7d6e000) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:2606
#6  0x0000560ded312616 in main (argc=<optimized out>, argv=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/ceph_osd.cc:633

Thread 7 (Thread 0x7f1b28499700 (LWP 17709)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f1b2da289ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x0)
    at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/x86_64-redhat-linux/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=this@entry=0x560df7bbb380, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:52
#3  0x0000560ded4549d9 in ceph::timer_detail::timer<ceph::time_detail::mono_clock>::timer_thread (this=0x560df7bbb318) at /home/ubuntu/src/markhpc/ceph/src/common/ceph_timer.h:142
#4  0x00007f1b2da2c230 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b28499700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 6 (Thread 0x7f1b27c98700 (LWP 17710)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x0000560ded945b9e in WaitUntil (when=..., mutex=..., this=0x560df7d6e080) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:64
#2  SafeTimer::timer_thread (this=0x560df7d6e070) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:110
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b27c98700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 5 (Thread 0x7f1b2ac4a700 (LWP 17703)):
#0  0x00007f1b2d195d13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000560dedaf8e64 in EpollDriver::event_wait (this=0x560df7b9f9b0, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/async/EventEpoll.cc:114
#2  0x0000560ded9e3cba in EventCenter::process_events (this=this@entry=0x560df7a19e80, timeout_microseconds=<optimized out>, timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7f1b2ac48930)
    at /home/ubuntu/src/markhpc/ceph/src/msg/async/Event.cc:395
#3  0x0000560ded9e6abe in NetworkStack::__lambda4::operator() (__closure=0x560df79d84d0) at /home/ubuntu/src/markhpc/ceph/src/msg/async/Stack.cc:51
#4  0x00007f1b2da2c230 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b2ac4a700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 4 (Thread 0x7f1b26c96700 (LWP 17712)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000560ded9458ff in Wait (mutex=..., this=0x560df7d704c8) at /home/ubuntu/src/markhpc/ceph/src/common/Cond.h:48
#2  SafeTimer::timer_thread (this=0x560df7d704b8) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:108
#3  0x0000560ded946f2d in SafeTimerThread::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/Timer.cc:30
#4  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b26c96700) at pthread_create.c:308
#5  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 3 (Thread 0x7f1b294d6700 (LWP 17707)):

#0  0x00007f1b2d18adfd in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000560ded932c53 in poll (__timeout=-1, __nfds=2, __fds=0x7f1b294d4970) at /usr/include/bits/poll2.h:46
#2  AdminSocket::entry (this=0x560df7b74000) at /home/ubuntu/src/markhpc/ceph/src/common/admin_socket.cc:250
#3  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b294d6700) at pthread_create.c:308
#4  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
---Type <return> to continue, or q <return> to quit---

Thread 2 (Thread 0x7f1b2b44b700 (LWP 17704)):
#0  0x00007f1b2d195d13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000560dedaf8e64 in EpollDriver::event_wait (this=0x560df7b31470, fired_events=std::vector of length 0, capacity 0, tvp=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/msg/async/EventEpoll.cc:114
#2  0x0000560ded9e3cba in EventCenter::process_events (this=this@entry=0x560df7a1a280, timeout_microseconds=<optimized out>, timeout_microseconds@entry=30000000, working_dur=working_dur@entry=0x7f1b2b449930)
    at /home/ubuntu/src/markhpc/ceph/src/msg/async/Event.cc:395
#3  0x0000560ded9e6abe in NetworkStack::__lambda4::operator() (__closure=0x560df79d84e0) at /home/ubuntu/src/markhpc/ceph/src/msg/async/Stack.cc:51
#4  0x00007f1b2da2c230 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#5  0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b2b44b700) at pthread_create.c:308
#6  0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x7f1b14c72700 (LWP 17763)):
#0  0x00007f1b2e0a923b in raise (sig=11) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x0000560ded90c426 in reraise_fatal (signum=11) at /home/ubuntu/src/markhpc/ceph/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11) at /home/ubuntu/src/markhpc/ceph/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  std::_Rb_tree<long, std::pair<long const, pg_pool_t>, std::_Select1st<std::pair<long const, pg_pool_t> >, std::less<long>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<long const, pg_pool_t> > >::find (
    this=this@entry=0x168, __k=@0x7f1b14c6ffe8: 1) at /usr/include/c++/4.8.2/bits/stl_tree.h:1803
#5  0x0000560deda0f1ea in find (__x=@0x7f1b14c6ffe8: 1, this=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_map.h:837
#6  get_pg_pool (p=1, this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/osd/OSDMap.h:1183
#7  OSDMap::_pg_to_up_acting_osds (this=this@entry=0x0, pg=..., up=up@entry=0x7f1b14c700a0, up_primary=up_primary@entry=0x0, acting=acting@entry=0x7f1b14c700c0, acting_primary=acting_primary@entry=0x0, 
    raw_pg_to_pg=raw_pg_to_pg@entry=false) at /home/ubuntu/src/markhpc/ceph/src/osd/OSDMap.cc:2105
#8  0x0000560ded426336 in OSDMap::is_up_acting_osd_shard (this=this@entry=0x0, pg=..., osd=10) at /home/ubuntu/src/markhpc/ceph/src/osd/OSDMap.h:1218
#9  0x0000560ded409955 in OSD::ShardedOpWQ::_process (this=0x560df7d6f240, thread_index=<optimized out>, hb=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/osd/OSD.cc:10067
#10 0x0000560ded94fdc9 in ShardedThreadPool::shardedthreadpool_worker (this=0x560df7d6ea30, thread_index=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.cc:343
#11 0x0000560ded951f50 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /home/ubuntu/src/markhpc/ceph/src/common/WorkQueue.h:689
#12 0x00007f1b2e0a1dc5 in start_thread (arg=0x7f1b14c72700) at pthread_create.c:308
#13 0x00007f1b2d19573d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

History

#1 Updated by Sage Weil over 6 years ago

  • Status changed from New to Fix Under Review

#2 Updated by xie xingguo over 6 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF