Actions
Bug #5392
closedosd: unfound objects from thrashing
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
ubuntu@plana41:~$ ceph health detail HEALTH_WARN 4 pgs recovering; 4 pgs stuck unclean; recovery 699/25994 degraded (2.689%); 134/12997 unfound (1.031%) pg 0.3b is stuck unclean for 32312.629395, current state active+recovering, last acting [3,5] pg 0.39 is stuck unclean for 32310.951536, current state active+recovering, last acting [5,3] pg 0.37 is stuck unclean for 32309.638516, current state active+recovering, last acting [5,3] pg 0.36 is stuck unclean for 32318.087019, current state active+recovering, last acting [3,5] pg 0.3b is active+recovering, acting [3,5], 50 unfound pg 0.39 is active+recovering, acting [5,3], 1 unfound pg 0.37 is active+recovering, acting [5,3], 46 unfound pg 0.36 is active+recovering, acting [3,5], 37 unfound recovery 699/25994 degraded (2.689%); 134/12997 unfound (1.031%) ubuntu@plana41:~$ ceph osd dump epoch 193 fsid 9f81f67c-6ad0-49e9-ae23-d60e7cb3beed created 2013-06-18 02:12:09.286514 modified 2013-06-18 03:11:56.787216 flags pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 34 last_change 63 owner 0 flags 1 crash_replay_interval 45 pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 24 pgp_num 24 last_change 1 owner 0 flags 1 pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 24 pgp_num 24 last_change 1 owner 0 flags 1 max_osd 6 osd.0 up out weight 0 up_from 33 up_thru 116 down_at 31 last_clean_interval [5,28) 10.214.132.37:6812/1720 10.214.132.37:6813/1720 10.214.132.37:6814/1720 10.214.132.37:6815/1720 exists,up 89a3a95b-1039-4f5f-ae8b-89b91e47965d osd.1 up out weight 0 up_from 5 up_thru 13 down_at 0 last_clean_interval [0,0) 10.214.132.37:6808/19825 10.214.132.37:6809/19825 10.214.132.37:6810/19825 10.214.132.37:6811/19825 exists,up f8a5ad3e-a9c2-4fbf-a6a8-80f81e8f7b58 osd.2 up out weight 0 up_from 2 up_thru 183 down_at 0 last_clean_interval [0,0) 10.214.132.37:6800/19821 10.214.132.37:6801/19821 10.214.132.37:6802/19821 10.214.132.37:6803/19821 exists,up c577c923-04e7-4bac-bab1-b9324d6f40e0 osd.3 up in weight 1 up_from 5 up_thru 192 down_at 0 last_clean_interval [0,0) 10.214.132.33:6801/17858 10.214.132.33:6803/17858 10.214.132.33:6805/17858 10.214.132.33:6807/17858 exists,up cbd7563d-5da4-40dc-b89f-ab8a2db76f34 osd.4 up out weight 0 up_from 3 up_thru 55 down_at 0 last_clean_interval [0,0) 10.214.132.33:6800/17859 10.214.132.33:6802/17859 10.214.132.33:6804/17859 10.214.132.33:6806/17859 exists,up e6d4f9ac-7ff8-490b-b738-ab59c6e5bbb0 osd.5 up in weight 1 up_from 10 up_thru 186 down_at 9 last_clean_interval [5,6) 10.214.132.33:6813/18438 10.214.132.33:6814/18438 10.214.132.33:6815/18438 10.214.132.33:6816/18438 exists,up 6620bae6-5e44-48b6-801d-017e536b2aba
job is
ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-06-18_01:00:05-rados-next-testing-basic/38759$ cat orig.config.yaml kernel: kdb: true sha1: dbb898fa64ead2446a8e7e40b90ab55b2e066e09 machine_type: plana nuke-on-error: true overrides: ceph: conf: global: ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject socket failures: 2500 mon: debug mon: 20 debug ms: 20 debug paxos: 20 fs: xfs log-whitelist: - slow request sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322 install: ceph: sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322 s3tests: branch: next workunit: sha1: df8a3e5591948dfd94de2e06640cfe54d2de4322 roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - client.0 tasks: - chef: null - clock.check: null - install: null - ceph: log-whitelist: - wrongly marked me down - objects unfound and apparently lost - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - radosbench: clients: - client.0 time: 1800
cluster is still stuck
Updated by Samuel Just almost 11 years ago
</pr#5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 45 (Thread 0x7f2d76424700 (LWP 15679)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65f60) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d65ea0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d76424700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 44 (Thread 0x7f2d7792b700 (LWP 15678)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65e40) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d65d80) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7792b700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 43 (Thread 0x7f2d8245f700 (LWP 15728)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x2315468) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2315280) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8245f700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 42 (Thread 0x7f2d75321700 (LWP 15729)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x2315968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2315780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d75321700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 41 (Thread 0x7f2d7a238700 (LWP 15733)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x2315280, buf=0x7f2d7a237dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x2315280) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a238700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 40 (Thread 0x7f2d7d043700 (LWP 15735)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0xb586c00) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0xb586b40) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7d043700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 39 (Thread 0x7f2d703fc700 (LWP 15737)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x2315780, buf=0x7f2d703fbdbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x2315780) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d703fc700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 38 (Thread 0x7f2d7c842700 (LWP 15739)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d65480) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7c842700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 37 (Thread 0x7f2d70c04700 (LWP 15776)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x4cd0e68) at ./common/Cond.h:55 #2 Pipe::writer (this=0x4cd0c80) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d70c04700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 36 (Thread 0x7f2d74d1b700 (LWP 15777)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x31d1968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x31d1780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d74d1b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 35 (Thread 0x7f2d9a214700 (LWP 15778)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x4cd0c80, buf=0x7f2d9a213dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x4cd0c80) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d9a214700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 34 (Thread 0x7f2d6fff8700 (LWP 15779)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x31d1780, buf=0x7f2d6fff7dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x31d1780) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fff8700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 33 (Thread 0x7f2d78e32700 (LWP 15780)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65780) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d656c0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d78e32700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 32 (Thread 0x7f2d72913700 (LWP 15781)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d650c0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d65000) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d72913700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 31 (Thread 0x7f2d70901700 (LWP 15823)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0xba37140) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x21b5500, pm=0x7f2d70900db0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x21b5500) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d70901700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 30 (Thread 0x7f2d6fef7700 (LWP 15824)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x21b56e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0x21b5500) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fef7700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 29 (Thread 0x7f2d6fdf6700 (LWP 15887)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x67bd800) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x227c780, pm=0x7f2d6fdf5db0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x227c780) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fdf6700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 28 (Thread 0x7f2d6fcf5700 (LWP 15888)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x227c968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x227c780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fcf5700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 27 (Thread 0x7f2d8215c700 (LWP 15898)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x5037780, buf=0x7f2d8215bdbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x5037780) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8215c700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 26 (Thread 0x7f2d7a137700 (LWP 15899)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x5f4c780, buf=0x7f2d7a136dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x5f4c780) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a137700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 25 (Thread 0x7f2d81b56700 (LWP 15900)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x5037968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x5037780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81b56700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 24 (Thread 0x7f2d7fd4d700 (LWP 15901)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d98e40) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d98d80) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7fd4d700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 23 (Thread 0x7f2d8205b700 (LWP 15902)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x5f4c968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x5f4c780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8205b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 22 (Thread 0x7f2d83562700 (LWP 15903)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0xb586e40) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0xb586d80) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d83562700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 21 (Thread 0x7f2d71109700 (LWP 15910)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0xd1a1c80, buf=0x7f2d71108dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0xd1a1c80) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d71109700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 20 (Thread 0x7f2d75220700 (LWP 15911)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x2272500, buf=0x7f2d7521fdbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x2272500) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d75220700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 19 (Thread 0x7f2d7f54c700 (LWP 15912)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0xd1a1e68) at ./common/Cond.h:55 #2 Pipe::writer (this=0xd1a1c80) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f54c700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 18 (Thread 0x7f2d7471c700 (LWP 15914)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x2037300) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x2037240) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7471c700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 17 (Thread 0x7f2d7f44b700 (LWP 15916)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x22726e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0x2272500) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f44b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f2d80e50700 (LWP 15919)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d98540) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d98480) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d80e50700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f2d7511f700 (LWP 15955)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0xbceb500, buf=0x7f2d7511edbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0xbceb500) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7511f700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f2d81752700 (LWP 15956)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x6798000, buf=0x7f2d81751dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x6798000) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81752700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 13 (Thread 0x7f2d70e06700 (LWP 15957)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0xbceb6e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0xbceb500) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d70e06700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 12 (Thread 0x7f2d7712a700 (LWP 15958)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x67981e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0x6798000) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7712a700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f2d75b22700 (LWP 15962)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0xb586300) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0xb586240) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d75b22700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 10 (Thread 0x7f2d79633700 (LWP 15961)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0xb586d20) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0xb586c60) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d79633700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 9 (Thread 0x7f2d6fbf4700 (LWP 16004)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x308e800) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x384d280, pm=0x7f2d6fbf3db0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x384d280) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6fbf4700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f2d6faf3700 (LWP 16005)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x384d468) at ./common/Cond.h:55 #2 Pipe::writer (this=0x384d280) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6faf3700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 7 (Thread 0x7f2d6f9f2700 (LWP 16036)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x6013e68) at ./common/Cond.h:55 #2 Pipe::writer (this=0x6013c80) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6f9f2700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f2d6f8f1700 (LWP 16037)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x38b9968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x38b9780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d6f8f1700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 5 (Thread 0x7f2d70800700 (LWP 16041)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x6013c80, buf=0x7f2d707ffdbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x6013c80) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d70800700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 4 (Thread 0x7f2d7f34a700 (LWP 16043)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x2037420) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x2037360) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7f34a700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 3 (Thread 0x7f2d70d05700 (LWP 16045)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000982342 in Pipe::tcp_read_wait (this=<optimized out>) at msg/Pipe.cc:2109 #2 0x0000000000982660 in Pipe::tcp_read (this=0x38b9780, buf=0x7f2d70d04dbf "\377", len=1) at msg/Pipe.cc:2082 #3 0x0000000000995814 in Pipe::reader (this=0x38b9780) at msg/Pipe.cc:1324 #4 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d70d05700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f2d73915700 (LWP 16047)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65660) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d655a0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d73915700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 ---Type <return> to continue, or q <return> to quit--- #5 0x0000000000000000 in ?? () Thread 1 (Thread 0x7f2d9a22e780 (LWP 17858)): #0 0x00007f2d99c0b148 in pthread_join (threadid=139833618441984, thread_return=0x0) at pthread_join.c:89 #1 0x000000000089bbb2 in Thread::join (this=0x1cee270, prval=<optimized out>) at common/Thread.cc:121 #2 0x0000000000894348 in SimpleMessenger::wait (this=0x1cee000) at msg/SimpleMessenger.cc:503 #3 0x00000000005c7c85 in main (argc=<optimized out>, argv=<optimized out>) at ceph_osd.cc:468 (gdb) thread apply all bt Thread 154 (Thread 0x7f2d97a8f700 (LWP 17864)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008110db in ceph::log::Log::entry (this=0x1cdb000) at log/Log.cc:323 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d97a8f700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 153 (Thread 0x7f2d9728e700 (LWP 17866)): #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:102 #1 0x00000000008bd3d8 in CephContextServiceThread::entry (this=0x1cd0300) at common/ceph_context.cc:57 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d9728e700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 152 (Thread 0x7f2d96a8d700 (LWP 17868)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x00000000008a845b in AdminSocket::entry (this=0x1cdb140) at common/admin_socket.cc:230 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d96a8d700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 151 (Thread 0x7f2d9628c700 (LWP 17896)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000007d8df1 in Wait (mutex=..., this=0x1cf4a40) at ./common/Cond.h:55 #2 WBThrottle::get_next_should_flush (this=0x1cf4948, next=0x7f2d9628bd30) at os/WBThrottle.cc:127 #3 0x00000000007d96ba in WBThrottle::entry (this=0x1cf4948) at os/WBThrottle.cc:145 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d9628c700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 150 (Thread 0x7f2d91a83700 (LWP 17897)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008955f1 in Wait (mutex=..., this=0x1cee4e0) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1cee000) at msg/SimpleMessenger.cc:206 #3 0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d91a83700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 149 (Thread 0x7f2d92284700 (LWP 17898)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008955f1 in Wait (mutex=..., this=0x1ceea60) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1cee580) at msg/SimpleMessenger.cc:206 #3 0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d92284700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 148 (Thread 0x7f2d92a85700 (LWP 17899)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008955f1 in Wait (mutex=..., this=0x1cefae0) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1cef600) at msg/SimpleMessenger.cc:206 #3 0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d92a85700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 147 (Thread 0x7f2d95a8b700 (LWP 17900)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008955f1 in Wait (mutex=..., this=0x1cf0060) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1cefb80) at msg/SimpleMessenger.cc:206 #3 0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d95a8b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 146 (Thread 0x7f2d9528a700 (LWP 17901)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008955f1 in Wait (mutex=..., this=0x1ceefe0) at ./common/Cond.h:55 #2 SimpleMessenger::reaper_entry (this=0x1ceeb00) at msg/SimpleMessenger.cc:206 #3 0x000000000089a48d in SimpleMessenger::ReaperThread::entry (this=<optimized out>) at msg/SimpleMessenger.h:422 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d9528a700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 145 (Thread 0x7f2d94a89700 (LWP 17902)): #0 0x00007f2d97d96033 in select () at ../sysdeps/unix/syscall-template.S:82 #1 0x00000000007ef714 in SignalHandler::entry (this=0x1d16300) at global/signal_handler.cc:210 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d94a89700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 144 (Thread 0x7f2d94288700 (LWP 17903)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a15ab in WaitUntil (when=..., mutex=..., this=0x1d11080) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x1d11070) at common/Timer.cc:114 #3 0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d94288700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 143 (Thread 0x7f2d93a87700 (LWP 17904)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008a1057 in Wait (mutex=..., this=0x1d12c80) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x1d12c70) at common/Timer.cc:112 #3 0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d93a87700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 142 (Thread 0x7f2d93286700 (LWP 17951)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x000000000078c3c4 in WaitUntil (when=..., mutex=..., this=0x1cf4638) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4638) at ./common/Cond.h:79 #3 FileStore::sync_entry (this=0x1cf4000) at os/FileStore.cc:3277 #4 0x00000000007a7b1d in FileStore::SyncThread::entry (this=<optimized out>) at os/FileStore.h:117 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d93286700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 141 (Thread 0x7f2d91282700 (LWP 17952)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000076c3f2 in Wait (mutex=..., this=0x1d09e38) at ./common/Cond.h:55 #2 FileJournal::write_thread_entry (this=0x1d09d00) at os/FileJournal.cc:1130 #3 0x000000000069fdbd in FileJournal::Writer::entry (this=<optimized out>) at ./os/FileJournal.h:334 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d91282700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 140 (Thread 0x7f2d90a81700 (LWP 17953)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000076350c in Wait (mutex=..., this=0x1d0a080) at ./common/Cond.h:55 #2 FileJournal::write_finish_thread_entry (this=0x1d09d00) at os/FileJournal.cc:1354 #3 0x000000000069fd9d in FileJournal::WriteFinisher::entry (this=<optimized out>) at ./os/FileJournal.h:344 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d90a81700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 139 (Thread 0x7f2d90280700 (LWP 17954)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008027f0 in Wait (mutex=..., this=0x1cf4078) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1cf4018) at common/Finisher.cc:80 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d90280700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 138 (Thread 0x7f2d8fa7f700 (LWP 17955)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1cf4da8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4da8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1cf4d30, wt=0x1cfb2a0) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8fa7f700) at pthread_create.c:308 ---Type <return> to continue, or q <return> to quit--- #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 137 (Thread 0x7f2d8f27e700 (LWP 17956)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1cf4da8) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1cf4da8) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1cf4d30, wt=0x1cfb2e0) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8f27e700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 136 (Thread 0x7f2d8ea7d700 (LWP 17957)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008027f0 in Wait (mutex=..., this=0x1cf4c60) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1cf4c00) at common/Finisher.cc:80 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ea7d700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 135 (Thread 0x7f2d8e27c700 (LWP 17958)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008027f0 in Wait (mutex=..., this=0x1cf4508) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1cf44a8) at common/Finisher.cc:80 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8e27c700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 134 (Thread 0x7f2d8da7b700 (LWP 17959)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008a1057 in Wait (mutex=..., this=0x1cf46e8) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x1cf46d8) at common/Timer.cc:112 #3 0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8da7b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 133 (Thread 0x7f2d8d27a700 (LWP 17984)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000096fbc8 in Wait (mutex=..., this=0x1cee150) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1cee0e8) at msg/DispatchQueue.cc:129 #3 0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8d27a700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 132 (Thread 0x7f2d8ca79700 (LWP 17985)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000953815 in Accepter::entry (this=0x1cee0c0) at msg/Accepter.cc:198 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ca79700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 131 (Thread 0x7f2d8c278700 (LWP 17986)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000096fbc8 in Wait (mutex=..., this=0x1ceec50) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1ceebe8) at msg/DispatchQueue.cc:129 #3 0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8c278700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 130 (Thread 0x7f2d8ba77700 (LWP 17987)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000953815 in Accepter::entry (this=0x1ceebc0) at msg/Accepter.cc:198 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8ba77700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 129 (Thread 0x7f2d8b276700 (LWP 17988)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000096fbc8 in Wait (mutex=..., this=0x1cee6d0) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1cee668) at msg/DispatchQueue.cc:129 #3 0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8b276700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 128 (Thread 0x7f2d8aa75700 (LWP 17989)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000096fbc8 in Wait (mutex=..., this=0x1cef750) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1cef6e8) at msg/DispatchQueue.cc:129 #3 0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8aa75700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 127 (Thread 0x7f2d8a274700 (LWP 17990)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000953815 in Accepter::entry (this=0x1cef6c0) at msg/Accepter.cc:198 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8a274700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 126 (Thread 0x7f2d89a73700 (LWP 17991)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000096fbc8 in Wait (mutex=..., this=0x1cefcd0) at ./common/Cond.h:55 #2 DispatchQueue::entry (this=0x1cefc68) at msg/DispatchQueue.cc:129 #3 0x0000000000899a2d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d89a73700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 125 (Thread 0x7f2d89272700 (LWP 17992)): #0 0x00007f2d97d91313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87 #1 0x0000000000953815 in Accepter::entry (this=0x1cefc40) at msg/Accepter.cc:198 #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d89272700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 124 (Thread 0x7f2d88a71700 (LWP 17993)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a15ab in WaitUntil (when=..., mutex=..., this=0x7fff5573e2c8) at common/Cond.h:71 #2 SafeTimer::timer_thread (this=0x7fff5573e2b8) at common/Timer.cc:114 #3 0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d88a71700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 123 (Thread 0x7f2d88270700 (LWP 17994)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008027f0 in Wait (mutex=..., this=0x7fff5573e3e0) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x7fff5573e380) at common/Finisher.cc:80 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d88270700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 122 (Thread 0x7f2d87a6f700 (LWP 17995)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d114d0) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d114d0) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1d11458, wt=0x1cfb5a0) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d87a6f700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 121 (Thread 0x7f2d8726e700 (LWP 17996)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d114d0) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d114d0) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1d11458, wt=0x1cfbae0) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 ---Type <return> to continue, or q <return> to quit--- #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8726e700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 120 (Thread 0x7f2d86a6d700 (LWP 17997)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d11658) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11658) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1d115e0, wt=0x1cfbba0) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d86a6d700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 119 (Thread 0x7f2d8626c700 (LWP 17998)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d117e0) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d117e0) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1d11768, wt=0x1cfba60) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8626c700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 118 (Thread 0x7f2d85a6b700 (LWP 17999)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x00000000008a465e in WaitUntil (when=..., mutex=..., this=0x1d11968) at common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11968) at common/Cond.h:79 #3 ThreadPool::worker (this=0x1d118f0, wt=0x1cfbb40) at common/WorkQueue.cc:137 #4 0x00000000008a6000 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:316 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d85a6b700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 117 (Thread 0x7f2d8526a700 (LWP 18000)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215 #1 0x0000000000671199 in WaitUntil (when=..., mutex=..., this=0x1d11b08) at ./common/Cond.h:71 #2 WaitInterval (interval=..., mutex=..., cct=<optimized out>, this=0x1d11b08) at ./common/Cond.h:79 #3 OSD::heartbeat_entry (this=0x1d11000) at osd/OSD.cc:2839 #4 0x00000000006b872d in OSD::T_Heartbeat::entry (this=<optimized out>) at osd/OSD.h:761 #5 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8526a700) at pthread_create.c:308 #6 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #7 0x0000000000000000 in ?? () Thread 116 (Thread 0x7f2d84a69700 (LWP 18001)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008027f0 in Wait (mutex=..., this=0x1d12df8) at ./common/Cond.h:55 #2 Finisher::finisher_thread_entry (this=0x1d12d98) at common/Finisher.cc:80 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d84a69700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 115 (Thread 0x7f2d84268700 (LWP 18002)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000008a1057 in Wait (mutex=..., this=0x1d12b58) at common/Cond.h:55 #2 SafeTimer::timer_thread (this=0x1d12b48) at common/Timer.cc:112 #3 0x00000000008a1f5d in SafeTimerThread::entry (this=<optimized out>) at common/Timer.cc:38 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d84268700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 114 (Thread 0x7f2d7a036700 (LWP 18185)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x1d62e68) at ./common/Cond.h:55 #2 Pipe::writer (this=0x1d62c80) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a036700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 113 (Thread 0x7f2d79f35700 (LWP 18186)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d65ae0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d65a20) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d79f35700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 112 (Thread 0x7f2d7a339700 (LWP 18307)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000995aca in Wait (mutex=..., this=0x1d62e68) at ./common/Cond.h:55 #2 Pipe::reader (this=0x1d62c80) at msg/Pipe.cc:1315 #3 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7a339700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 111 (Thread 0x7f2d81651700 (LWP 19250)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000995aca in Wait (mutex=..., this=0x1f0b968) at ./common/Cond.h:55 #2 Pipe::reader (this=0x1f0b780) at msg/Pipe.cc:1315 #3 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81651700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 110 (Thread 0x7f2d76f28700 (LWP 19252)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x1f0b968) at ./common/Cond.h:55 #2 Pipe::writer (this=0x1f0b780) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d76f28700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 109 (Thread 0x7f2d76e27700 (LWP 19253)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1d98660) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1d985a0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d76e27700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 108 (Thread 0x7f2d7812c700 (LWP 19334)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00000000009bbd63 in leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper(void*) () #2 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7812c700) at pthread_create.c:308 #3 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #4 0x0000000000000000 in ?? () Thread 107 (Thread 0x7f2d73f1b700 (LWP 22990)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000995aca in Wait (mutex=..., this=0x6c2c6e8) at ./common/Cond.h:55 #2 Pipe::reader (this=0x6c2c500) at msg/Pipe.cc:1315 #3 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d73f1b700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 106 (Thread 0x7f2d73e1a700 (LWP 22991)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x6c2c6e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0x6c2c500) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d73e1a700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 105 (Thread 0x7f2d73114700 (LWP 22992)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x2037660) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x20375a0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d73114700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 ---Type <return> to continue, or q <return> to quit--- #5 0x0000000000000000 in ?? () Thread 104 (Thread 0x7f2d76626700 (LWP 23073)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000995aca in Wait (mutex=..., this=0xba191e8) at ./common/Cond.h:55 #2 Pipe::reader (this=0xba19000) at msg/Pipe.cc:1315 #3 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d76626700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 103 (Thread 0x7f2d8124e700 (LWP 23074)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0xba191e8) at ./common/Cond.h:55 #2 Pipe::writer (this=0xba19000) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8124e700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 102 (Thread 0x7f2d7bf40700 (LWP 23075)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1eb18a0) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1eb17e0) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7bf40700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 101 (Thread 0x7f2d81d58700 (LWP 23233)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x0000000000995aca in Wait (mutex=..., this=0x3fa7be8) at ./common/Cond.h:55 #2 Pipe::reader (this=0x3fa7a00) at msg/Pipe.cc:1315 #3 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81d58700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 100 (Thread 0x7f2d81f5a700 (LWP 23235)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000098e45d in Wait (mutex=..., this=0x3fa7be8) at ./common/Cond.h:55 #2 Pipe::writer (this=0x3fa7a00) at msg/Pipe.cc:1575 #3 0x000000000099876d in Pipe::Writer::entry (this=<optimized out>) at msg/Pipe.h:59 #4 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81f5a700) at pthread_create.c:308 #5 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #6 0x0000000000000000 in ?? () Thread 99 (Thread 0x7f2d72112700 (LWP 23236)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000097f0ff in Wait (mutex=..., this=0x1eb1300) at ./common/Cond.h:55 #2 Pipe::DelayedDelivery::entry (this=0x1eb1240) at msg/Pipe.cc:193 #3 0x00007f2d99c09e9a in start_thread (arg=0x7f2d72112700) at pthread_create.c:308 #4 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #5 0x0000000000000000 in ?? () Thread 98 (Thread 0x7f2d81954700 (LWP 2925)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x308e280) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x4cae780, pm=0x7f2d81953db0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x4cae780) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d81954700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 97 (Thread 0x7f2d82560700 (LWP 3497)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x6b6de80) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0xb587a00, pm=0x7f2d8255fdb0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0xb587a00) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d82560700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 96 (Thread 0x7f2d8235e700 (LWP 4248)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x389c3c0) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x223f280, pm=0x7f2d8235ddb0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x223f280) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d8235e700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 95 (Thread 0x7f2d7d245700 (LWP 4884)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x2211700) at ./common/Cond.h:55 #2 Throttle::_wait (this=0x1cca320, c=1) at common/Throttle.cc:99 #3 0x000000000089d091 in Throttle::get (this=0x1cca320, c=1, m=0) at common/Throttle.cc:157 #4 0x0000000000983274 in Pipe::read_message (this=0x223fc80, pm=0x7f2d7d244db0) at msg/Pipe.cc:1671 #5 0x0000000000995fe2 in Pipe::reader (this=0x223fc80) at msg/Pipe.cc:1355 #6 0x000000000099878d in Pipe::Reader::entry (this=<optimized out>) at msg/Pipe.h:47 #7 0x00007f2d99c09e9a in start_thread (arg=0x7f2d7d245700) at pthread_create.c:308 #8 0x00007f2d97d9cccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #9 0x0000000000000000 in ?? () Thread 94 (Thread 0x7f2d77029700 (LWP 5337)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x000000000089c3c7 in Wait (mutex=..., this=0x3f42f40) at ./co
Updated by Samuel Just almost 11 years ago
There seem to be a lot of threads waiting on throttle. Unfortunately, the test timed out before I could get more info (no coredump).
Updated by Sage Weil almost 11 years ago
ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-07-08_20:00:15-rados-cuttlefish-testing-basic/59715
machines are currently sitting idle with 32 unfound objects
Updated by Samuel Just almost 11 years ago
- Status changed from 12 to 7
This could be explained by a failure to resurrect a parent pg causing recovery on an hobject to perform writes on an object left in a deleting parent pg collection.
Actions