Project

General

Profile

Actions
Copy link

Bug #5392

closed

osd: unfound objects from thrashing

Added by Sage Weil almost 11 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@plana41:~$ ceph health detail
HEALTH_WARN 4 pgs recovering; 4 pgs stuck unclean; recovery 699/25994 degraded (2.689%); 134/12997 unfound (1.031%)
pg 0.3b is stuck unclean for 32312.629395, current state active+recovering, last acting [3,5]
pg 0.39 is stuck unclean for 32310.951536, current state active+recovering, last acting [5,3]
pg 0.37 is stuck unclean for 32309.638516, current state active+recovering, last acting [5,3]
pg 0.36 is stuck unclean for 32318.087019, current state active+recovering, last acting [3,5]
pg 0.3b is active+recovering, acting [3,5], 50 unfound
pg 0.39 is active+recovering, acting [5,3], 1 unfound
pg 0.37 is active+recovering, acting [5,3], 46 unfound
pg 0.36 is active+recovering, acting [3,5], 37 unfound
recovery 699/25994 degraded (2.689%); 134/12997 unfound (1.031%)
ubuntu@plana41:~$ ceph osd dump
epoch 193
fsid 9f81f67c-6ad0-49e9-ae23-d60e7cb3beed
created 2013-06-18 02:12:09.286514
modified 2013-06-18 03:11:56.787216
flags 

pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 34 last_change 63 owner 0 flags 1 crash_replay_interval 45
pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 24 pgp_num 24 last_change 1 owner 0 flags 1
pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 24 pgp_num 24 last_change 1 owner 0 flags 1

max_osd 6
osd.0 up   out weight 0 up_from 33 up_thru 116 down_at 31 last_clean_interval [5,28) 10.214.132.37:6812/1720 10.214.132.37:6813/1720 10.214.132.37:6814/1720 10.214.132.37:6815/1720 exists,up 89a3a95b-1039-4f5f-ae8b-89b91e47965d
osd.1 up   out weight 0 up_from 5 up_thru 13 down_at 0 last_clean_interval [0,0) 10.214.132.37:6808/19825 10.214.132.37:6809/19825 10.214.132.37:6810/19825 10.214.132.37:6811/19825 exists,up f8a5ad3e-a9c2-4fbf-a6a8-80f81e8f7b58
osd.2 up   out weight 0 up_from 2 up_thru 183 down_at 0 last_clean_interval [0,0) 10.214.132.37:6800/19821 10.214.132.37:6801/19821 10.214.132.37:6802/19821 10.214.132.37:6803/19821 exists,up c577c923-04e7-4bac-bab1-b9324d6f40e0
osd.3 up   in  weight 1 up_from 5 up_thru 192 down_at 0 last_clean_interval [0,0) 10.214.132.33:6801/17858 10.214.132.33:6803/17858 10.214.132.33:6805/17858 10.214.132.33:6807/17858 exists,up cbd7563d-5da4-40dc-b89f-ab8a2db76f34
osd.4 up   out weight 0 up_from 3 up_thru 55 down_at 0 last_clean_interval [0,0) 10.214.132.33:6800/17859 10.214.132.33:6802/17859 10.214.132.33:6804/17859 10.214.132.33:6806/17859 exists,up e6d4f9ac-7ff8-490b-b738-ab59c6e5bbb0
osd.5 up   in  weight 1 up_from 10 up_thru 186 down_at 9 last_clean_interval [5,6) 10.214.132.33:6813/18438 10.214.132.33:6814/18438 10.214.132.33:6815/18438 10.214.132.33:6816/18438 exists,up 6620bae6-5e44-48b6-801d-017e536b2aba

job is
ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-06-18_01:00:05-rados-next-testing-basic/38759$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: dbb898fa64ead2446a8e7e40b90ab55b2e066e09
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject socket failures: 2500
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
    fs: xfs
    log-whitelist:
    - slow request
    sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322
  install:
    ceph:
      sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322
  s3tests:
    branch: next
  workunit:
    sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- radosbench:
    clients:
    - client.0
    time: 1800

cluster is still stuck

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #5269: osd: EEXIST on mkcollResolved06/06/2013

Actions
Actions
Copy link
 #1

Updated by Sage Weil almost 11 years ago

  • Assignee set to Samuel Just
Actions
Copy link
 #2

Updated by Samuel Just almost 11 years ago 


</pr#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 45 (Thread 0x7f2d76424700 (LWP 15679)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65f60) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d65ea0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d76424700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 44 (Thread 0x7f2d7792b700 (LWP 15678)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65e40) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d65d80) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7792b700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 43 (Thread 0x7f2d8245f700 (LWP 15728)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x2315468) at ./common/Cond.h:55
#2  Pipe::writer (this=0x2315280) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8245f700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 42 (Thread 0x7f2d75321700 (LWP 15729)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x2315968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x2315780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d75321700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 41 (Thread 0x7f2d7a238700 (LWP 15733)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x2315280, buf=0x7f2d7a237dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x2315280) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a238700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 40 (Thread 0x7f2d7d043700 (LWP 15735)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0xb586c00) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0xb586b40) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7d043700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 39 (Thread 0x7f2d703fc700 (LWP 15737)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x2315780, buf=0x7f2d703fbdbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x2315780) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d703fc700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 38 (Thread 0x7f2d7c842700 (LWP 15739)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d65480) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7c842700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 37 (Thread 0x7f2d70c04700 (LWP 15776)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x4cd0e68) at ./common/Cond.h:55
#2  Pipe::writer (this=0x4cd0c80) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d70c04700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 36 (Thread 0x7f2d74d1b700 (LWP 15777)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x31d1968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x31d1780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d74d1b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 35 (Thread 0x7f2d9a214700 (LWP 15778)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x4cd0c80, buf=0x7f2d9a213dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x4cd0c80) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d9a214700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 34 (Thread 0x7f2d6fff8700 (LWP 15779)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x31d1780, buf=0x7f2d6fff7dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x31d1780) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fff8700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 33 (Thread 0x7f2d78e32700 (LWP 15780)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65780) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d656c0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d78e32700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 32 (Thread 0x7f2d72913700 (LWP 15781)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d650c0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d65000) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d72913700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 31 (Thread 0x7f2d70901700 (LWP 15823)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0xba37140) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x21b5500, pm=0x7f2d70900db0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x21b5500) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d70901700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 30 (Thread 0x7f2d6fef7700 (LWP 15824)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x21b56e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x21b5500) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fef7700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 29 (Thread 0x7f2d6fdf6700 (LWP 15887)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x67bd800) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x227c780, pm=0x7f2d6fdf5db0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x227c780) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fdf6700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7f2d6fcf5700 (LWP 15888)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x227c968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x227c780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fcf5700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 27 (Thread 0x7f2d8215c700 (LWP 15898)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x5037780, buf=0x7f2d8215bdbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x5037780) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8215c700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 26 (Thread 0x7f2d7a137700 (LWP 15899)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x5f4c780, buf=0x7f2d7a136dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x5f4c780) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a137700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 25 (Thread 0x7f2d81b56700 (LWP 15900)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x5037968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x5037780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81b56700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 24 (Thread 0x7f2d7fd4d700 (LWP 15901)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d98e40) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d98d80) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7fd4d700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7f2d8205b700 (LWP 15902)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x5f4c968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x5f4c780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8205b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 22 (Thread 0x7f2d83562700 (LWP 15903)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0xb586e40) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0xb586d80) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d83562700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 21 (Thread 0x7f2d71109700 (LWP 15910)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0xd1a1c80, buf=0x7f2d71108dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0xd1a1c80) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d71109700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f2d75220700 (LWP 15911)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x2272500, buf=0x7f2d7521fdbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x2272500) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d75220700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f2d7f54c700 (LWP 15912)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0xd1a1e68) at ./common/Cond.h:55
#2  Pipe::writer (this=0xd1a1c80) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f54c700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f2d7471c700 (LWP 15914)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x2037300) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x2037240) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7471c700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 17 (Thread 0x7f2d7f44b700 (LWP 15916)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x22726e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x2272500) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f44b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f2d80e50700 (LWP 15919)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d98540) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d98480) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d80e50700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f2d7511f700 (LWP 15955)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0xbceb500, buf=0x7f2d7511edbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0xbceb500) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7511f700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f2d81752700 (LWP 15956)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x6798000, buf=0x7f2d81751dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x6798000) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81752700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f2d70e06700 (LWP 15957)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0xbceb6e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0xbceb500) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d70e06700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f2d7712a700 (LWP 15958)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x67981e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x6798000) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7712a700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f2d75b22700 (LWP 15962)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0xb586300) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0xb586240) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d75b22700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f2d79633700 (LWP 15961)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0xb586d20) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0xb586c60) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d79633700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f2d6fbf4700 (LWP 16004)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x308e800) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x384d280, pm=0x7f2d6fbf3db0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x384d280) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fbf4700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f2d6faf3700 (LWP 16005)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x384d468) at ./common/Cond.h:55
#2  Pipe::writer (this=0x384d280) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6faf3700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f2d6f9f2700 (LWP 16036)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x6013e68) at ./common/Cond.h:55
#2  Pipe::writer (this=0x6013c80) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6f9f2700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f2d6f8f1700 (LWP 16037)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x38b9968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x38b9780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d6f8f1700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f2d70800700 (LWP 16041)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x6013c80, buf=0x7f2d707ffdbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x6013c80) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d70800700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f2d7f34a700 (LWP 16043)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x2037420) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x2037360) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f34a700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f2d70d05700 (LWP 16045)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109
#2  0x0000000000982660 in Pipe::tcp_read (this=0x38b9780, buf=0x7f2d70d04dbf "\377", len=1) at msg/Pipe.cc:2082
#3  0x0000000000995814 in Pipe::reader (this=0x38b9780) at msg/Pipe.cc:1324
#4  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d70d05700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f2d73915700 (LWP 16047)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65660) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d655a0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d73915700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
---Type <return> to continue, or q <return> to quit---
#5  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f2d9a22e780 (LWP 17858)):
#0  0x00007f2d99c0b148 in pthread_join (threadid=139833618441984, thread_return=0x0) at pthread_join.c:89
#1  0x000000000089bbb2 in Thread::join (this=0x1cee270, prval=<optimized out>) at common/Thread.cc:121
#2  0x0000000000894348 in SimpleMessenger::wait (this=0x1cee000) at msg/SimpleMessenger.cc:503
#3  0x00000000005c7c85 in main (argc=<optimized out>, argv=<optimized out>) at ceph_osd.cc:468
(gdb) thread apply all bt

Thread 154 (Thread 0x7f2d97a8f700 (LWP 17864)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008110db in ceph::log::Log::entry (this=0x1cdb000) at log/Log.cc:323
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d97a8f700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 153 (Thread 0x7f2d9728e700 (LWP 17866)):
#0  sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102
#1  0x00000000008bd3d8 in CephContextServiceThread::entry (this=0x1cd0300) at common/ceph_context.cc:57
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d9728e700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 152 (Thread 0x7f2d96a8d700 (LWP 17868)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x00000000008a845b in AdminSocket::entry (this=0x1cdb140) at common/admin_socket.cc:230
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d96a8d700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 151 (Thread 0x7f2d9628c700 (LWP 17896)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000007d8df1 in Wait (mutex=..., this=0x1cf4a40) at ./common/Cond.h:55
#2  WBThrottle::get_next_should_flush (this=0x1cf4948, next=0x7f2d9628bd30) at os/WBThrottle.cc:127
#3  0x00000000007d96ba in WBThrottle::entry (this=0x1cf4948) at os/WBThrottle.cc:145
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d9628c700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 150 (Thread 0x7f2d91a83700 (LWP 17897)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008955f1 in Wait (mutex=..., this=0x1cee4e0) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x1cee000) at msg/SimpleMessenger.cc:206
#3  0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d91a83700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 149 (Thread 0x7f2d92284700 (LWP 17898)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008955f1 in Wait (mutex=..., this=0x1ceea60) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x1cee580) at msg/SimpleMessenger.cc:206
#3  0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d92284700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 148 (Thread 0x7f2d92a85700 (LWP 17899)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008955f1 in Wait (mutex=..., this=0x1cefae0) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x1cef600) at msg/SimpleMessenger.cc:206
#3  0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d92a85700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 147 (Thread 0x7f2d95a8b700 (LWP 17900)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008955f1 in Wait (mutex=..., this=0x1cf0060) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x1cefb80) at msg/SimpleMessenger.cc:206
#3  0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d95a8b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 146 (Thread 0x7f2d9528a700 (LWP 17901)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008955f1 in Wait (mutex=..., this=0x1ceefe0) at ./common/Cond.h:55
#2  SimpleMessenger::reaper_entry (this=0x1ceeb00) at msg/SimpleMessenger.cc:206
#3  0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d9528a700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 145 (Thread 0x7f2d94a89700 (LWP 17902)):
#0  0x00007f2d97d96033 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x00000000007ef714 in SignalHandler::entry (this=0x1d16300) at global/signal_handler.cc:210
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d94a89700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 144 (Thread 0x7f2d94288700 (LWP 17903)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a15ab in WaitUntil (when=..., mutex=..., this=0x1d11080) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x1d11070) at common/Timer.cc:114
#3  0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d94288700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 143 (Thread 0x7f2d93a87700 (LWP 17904)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008a1057 in Wait (mutex=..., this=0x1d12c80) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x1d12c70) at common/Timer.cc:112
#3  0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d93a87700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 142 (Thread 0x7f2d93286700 (LWP 17951)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x000000000078c3c4 in WaitUntil (when=..., mutex=..., this=0x1cf4638) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4638) at ./common/Cond.h:79
#3  FileStore::sync_entry (this=0x1cf4000) at os/FileStore.cc:3277
#4  0x00000000007a7b1d in FileStore::SyncThread::entry (this=<optimized out>) at os/FileStore.h:117
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d93286700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 141 (Thread 0x7f2d91282700 (LWP 17952)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000076c3f2 in Wait (mutex=..., this=0x1d09e38) at ./common/Cond.h:55
#2  FileJournal::write_thread_entry (this=0x1d09d00) at os/FileJournal.cc:1130
#3  0x000000000069fdbd in FileJournal::Writer::entry (this=<optimized out>) at ./os/FileJournal.h:334
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d91282700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 140 (Thread 0x7f2d90a81700 (LWP 17953)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000076350c in Wait (mutex=..., this=0x1d0a080) at ./common/Cond.h:55
#2  FileJournal::write_finish_thread_entry (this=0x1d09d00) at os/FileJournal.cc:1354
#3  0x000000000069fd9d in FileJournal::WriteFinisher::entry (this=<optimized out>) at ./os/FileJournal.h:344
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d90a81700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 139 (Thread 0x7f2d90280700 (LWP 17954)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008027f0 in Wait (mutex=..., this=0x1cf4078) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x1cf4018) at common/Finisher.cc:80
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d90280700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 138 (Thread 0x7f2d8fa7f700 (LWP 17955)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1cf4da8) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4da8) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1cf4d30, wt=0x1cfb2a0) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8fa7f700) at pthread_create.c:308
---Type <return> to continue, or q <return> to quit---
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 137 (Thread 0x7f2d8f27e700 (LWP 17956)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1cf4da8) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4da8) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1cf4d30, wt=0x1cfb2e0) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8f27e700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 136 (Thread 0x7f2d8ea7d700 (LWP 17957)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008027f0 in Wait (mutex=..., this=0x1cf4c60) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x1cf4c00) at common/Finisher.cc:80
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ea7d700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 135 (Thread 0x7f2d8e27c700 (LWP 17958)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008027f0 in Wait (mutex=..., this=0x1cf4508) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x1cf44a8) at common/Finisher.cc:80
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8e27c700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 134 (Thread 0x7f2d8da7b700 (LWP 17959)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008a1057 in Wait (mutex=..., this=0x1cf46e8) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x1cf46d8) at common/Timer.cc:112
#3  0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8da7b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 133 (Thread 0x7f2d8d27a700 (LWP 17984)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000096fbc8 in Wait (mutex=..., this=0x1cee150) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x1cee0e8) at msg/DispatchQueue.cc:129
#3  0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8d27a700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 132 (Thread 0x7f2d8ca79700 (LWP 17985)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000953815 in Accepter::entry (this=0x1cee0c0) at msg/Accepter.cc:198
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ca79700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 131 (Thread 0x7f2d8c278700 (LWP 17986)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000096fbc8 in Wait (mutex=..., this=0x1ceec50) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x1ceebe8) at msg/DispatchQueue.cc:129
#3  0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8c278700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 130 (Thread 0x7f2d8ba77700 (LWP 17987)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000953815 in Accepter::entry (this=0x1ceebc0) at msg/Accepter.cc:198
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ba77700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 129 (Thread 0x7f2d8b276700 (LWP 17988)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000096fbc8 in Wait (mutex=..., this=0x1cee6d0) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x1cee668) at msg/DispatchQueue.cc:129
#3  0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8b276700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 128 (Thread 0x7f2d8aa75700 (LWP 17989)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000096fbc8 in Wait (mutex=..., this=0x1cef750) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x1cef6e8) at msg/DispatchQueue.cc:129
#3  0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8aa75700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 127 (Thread 0x7f2d8a274700 (LWP 17990)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000953815 in Accepter::entry (this=0x1cef6c0) at msg/Accepter.cc:198
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8a274700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 126 (Thread 0x7f2d89a73700 (LWP 17991)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000096fbc8 in Wait (mutex=..., this=0x1cefcd0) at ./common/Cond.h:55
#2  DispatchQueue::entry (this=0x1cefc68) at msg/DispatchQueue.cc:129
#3  0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d89a73700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 125 (Thread 0x7f2d89272700 (LWP 17992)):
#0  0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000000000953815 in Accepter::entry (this=0x1cefc40) at msg/Accepter.cc:198
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d89272700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 124 (Thread 0x7f2d88a71700 (LWP 17993)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a15ab in WaitUntil (when=..., mutex=..., this=0x7fff5573e2c8) at common/Cond.h:71
#2  SafeTimer::timer_thread (this=0x7fff5573e2b8) at common/Timer.cc:114
#3  0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d88a71700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 123 (Thread 0x7f2d88270700 (LWP 17994)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008027f0 in Wait (mutex=..., this=0x7fff5573e3e0) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x7fff5573e380) at common/Finisher.cc:80
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d88270700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 122 (Thread 0x7f2d87a6f700 (LWP 17995)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d114d0) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d114d0) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1d11458, wt=0x1cfb5a0) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d87a6f700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 121 (Thread 0x7f2d8726e700 (LWP 17996)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d114d0) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d114d0) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1d11458, wt=0x1cfbae0) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
---Type <return> to continue, or q <return> to quit---
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8726e700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 120 (Thread 0x7f2d86a6d700 (LWP 17997)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d11658) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11658) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1d115e0, wt=0x1cfbba0) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d86a6d700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 119 (Thread 0x7f2d8626c700 (LWP 17998)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d117e0) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d117e0) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1d11768, wt=0x1cfba60) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8626c700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 118 (Thread 0x7f2d85a6b700 (LWP 17999)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d11968) at common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11968) at common/Cond.h:79
#3  ThreadPool::worker (this=0x1d118f0, wt=0x1cfbb40) at common/WorkQueue.cc:137
#4  0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d85a6b700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 117 (Thread 0x7f2d8526a700 (LWP 18000)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1  0x0000000000671199 in WaitUntil (when=..., mutex=..., this=0x1d11b08) at ./common/Cond.h:71
#2  WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11b08) at ./common/Cond.h:79
#3  OSD::heartbeat_entry (this=0x1d11000) at osd/OSD.cc:2839
#4  0x00000000006b872d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:761
#5  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8526a700) at pthread_create.c:308
#6  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7  0x0000000000000000 in ?? ()

Thread 116 (Thread 0x7f2d84a69700 (LWP 18001)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008027f0 in Wait (mutex=..., this=0x1d12df8) at ./common/Cond.h:55
#2  Finisher::finisher_thread_entry (this=0x1d12d98) at common/Finisher.cc:80
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d84a69700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 115 (Thread 0x7f2d84268700 (LWP 18002)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000008a1057 in Wait (mutex=..., this=0x1d12b58) at common/Cond.h:55
#2  SafeTimer::timer_thread (this=0x1d12b48) at common/Timer.cc:112
#3  0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d84268700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 114 (Thread 0x7f2d7a036700 (LWP 18185)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x1d62e68) at ./common/Cond.h:55
#2  Pipe::writer (this=0x1d62c80) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a036700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 113 (Thread 0x7f2d79f35700 (LWP 18186)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d65ae0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d65a20) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d79f35700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 112 (Thread 0x7f2d7a339700 (LWP 18307)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000995aca in Wait (mutex=..., this=0x1d62e68) at ./common/Cond.h:55
#2  Pipe::reader (this=0x1d62c80) at msg/Pipe.cc:1315
#3  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a339700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 111 (Thread 0x7f2d81651700 (LWP 19250)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000995aca in Wait (mutex=..., this=0x1f0b968) at ./common/Cond.h:55
#2  Pipe::reader (this=0x1f0b780) at msg/Pipe.cc:1315
#3  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81651700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 110 (Thread 0x7f2d76f28700 (LWP 19252)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x1f0b968) at ./common/Cond.h:55
#2  Pipe::writer (this=0x1f0b780) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d76f28700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 109 (Thread 0x7f2d76e27700 (LWP 19253)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1d98660) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1d985a0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d76e27700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 108 (Thread 0x7f2d7812c700 (LWP 19334)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00000000009bbd63 in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper(void*) ()
#2  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7812c700) at pthread_create.c:308
#3  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()

Thread 107 (Thread 0x7f2d73f1b700 (LWP 22990)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000995aca in Wait (mutex=..., this=0x6c2c6e8) at ./common/Cond.h:55
#2  Pipe::reader (this=0x6c2c500) at msg/Pipe.cc:1315
#3  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d73f1b700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 106 (Thread 0x7f2d73e1a700 (LWP 22991)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x6c2c6e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x6c2c500) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d73e1a700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 105 (Thread 0x7f2d73114700 (LWP 22992)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x2037660) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x20375a0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d73114700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
---Type <return> to continue, or q <return> to quit---
#5  0x0000000000000000 in ?? ()

Thread 104 (Thread 0x7f2d76626700 (LWP 23073)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000995aca in Wait (mutex=..., this=0xba191e8) at ./common/Cond.h:55
#2  Pipe::reader (this=0xba19000) at msg/Pipe.cc:1315
#3  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d76626700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 103 (Thread 0x7f2d8124e700 (LWP 23074)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0xba191e8) at ./common/Cond.h:55
#2  Pipe::writer (this=0xba19000) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8124e700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 102 (Thread 0x7f2d7bf40700 (LWP 23075)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1eb18a0) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1eb17e0) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7bf40700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 101 (Thread 0x7f2d81d58700 (LWP 23233)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x0000000000995aca in Wait (mutex=..., this=0x3fa7be8) at ./common/Cond.h:55
#2  Pipe::reader (this=0x3fa7a00) at msg/Pipe.cc:1315
#3  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81d58700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 100 (Thread 0x7f2d81f5a700 (LWP 23235)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000098e45d in Wait (mutex=..., this=0x3fa7be8) at ./common/Cond.h:55
#2  Pipe::writer (this=0x3fa7a00) at msg/Pipe.cc:1575
#3  0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59
#4  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81f5a700) at pthread_create.c:308
#5  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#6  0x0000000000000000 in ?? ()

Thread 99 (Thread 0x7f2d72112700 (LWP 23236)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000097f0ff in Wait (mutex=..., this=0x1eb1300) at ./common/Cond.h:55
#2  Pipe::DelayedDelivery::entry (this=0x1eb1240) at msg/Pipe.cc:193
#3  0x00007f2d99c09e9a in start_thread (arg=0x7f2d72112700) at pthread_create.c:308
#4  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5  0x0000000000000000 in ?? ()

Thread 98 (Thread 0x7f2d81954700 (LWP 2925)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x308e280) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x4cae780, pm=0x7f2d81953db0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x4cae780) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d81954700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 97 (Thread 0x7f2d82560700 (LWP 3497)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x6b6de80) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0xb587a00, pm=0x7f2d8255fdb0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0xb587a00) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d82560700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 96 (Thread 0x7f2d8235e700 (LWP 4248)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x389c3c0) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x223f280, pm=0x7f2d8235ddb0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x223f280) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d8235e700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 95 (Thread 0x7f2d7d245700 (LWP 4884)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x2211700) at ./common/Cond.h:55
#2  Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99
#3  0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157
#4  0x0000000000983274 in Pipe::read_message (this=0x223fc80, pm=0x7f2d7d244db0) at msg/Pipe.cc:1671
#5  0x0000000000995fe2 in Pipe::reader (this=0x223fc80) at msg/Pipe.cc:1355
#6  0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47
#7  0x00007f2d99c09e9a in start_thread (arg=0x7f2d7d245700) at pthread_create.c:308
#8  0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9  0x0000000000000000 in ?? ()

Thread 94 (Thread 0x7f2d77029700 (LWP 5337)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x000000000089c3c7 in Wait (mutex=..., this=0x3f42f40) at ./co
Actions
Copy link
 #3

Updated by Samuel Just almost 11 years ago

There seem to be a lot of threads waiting on throttle. Unfortunately, the test timed out before I could get more info (no coredump).

Actions
Copy link
 #4

Updated by Sage Weil almost 11 years ago

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-07-08_20:00:15-rados-cuttlefish-testing-basic/59715

machines are currently sitting idle with 32 unfound objects

Actions
Copy link
 #5

Updated by Samuel Just almost 11 years ago

trying to reproduce with logging.

Actions
Copy link
 #6

Updated by Samuel Just over 10 years ago

  • Status changed from 12 to 7

This could be explained by a failure to resurrect a parent pg causing recovery on an hobject to perform writes on an object left in a deleting parent pg collection.

Actions
Copy link
 #7

Updated by Samuel Just over 10 years ago

  • Assignee deleted (Samuel Just)
Actions
Copy link
 #8

Updated by Samuel Just over 10 years ago

  • Status changed from 7 to Resolved
Actions
Copy link

Also available in: Atom PDF