Project

General

Profile

Bug #909

ceph-client+ceph v0.25.1,iozone test, "libceph: tid 115358 timed out on osd0, will reset osd"

Added by changping Wu about 13 years ago. Updated almost 13 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
libceph
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Hi ,
I am doing iozone test for
ceph v0.25.1 + ceph-client master , commit :61e062a18f10f57fb507f4d883e7d1fce898a8a7
but printk many "osd timeout ",so the performance is very slow.this issue is very easy to reproduce.

the reproducing steps is at the below:

1. build a ceph 0.25.1 server with the attachment file ceph.conf + ext4/btrfs;
2. git ceph-client :commit :61e062a18f10f57fb507f4d883e7d1fce898a8a7,make menuconfig&&make -j10 &&make modules_install && make install && reboot;
3. mount.ceph monaddr:6789:/ /mnt/ceph -o mount_timeout=10
4. run iozone:
$iozone -z -c -e -a -n 200M -g 2000M -i 0 -i 1 -i 2 -f /mnt/ceph/fio -Rb ./iozone.xls

------------------------
I note that "osd timeout" issue always exits at each version of ceph-client/[-standalone] when do stress test.
which decrease the R/W performance and some times, make ceph client host hang up .

I have some of the ideas for this issue, maybe,they are useful for fixing the bug.

1).prolong timeout time ;
2).add a timer for each of the R/W requests ,not osd ;
3).if R/W stress is too strong ,abort all of the requests,
for instance, re-send timeout requests , but still fail,then abort all of the R/W requests,
make sure that ceph client system doesn't become very slow or hang up .

Jeff Wu

-----------------------
logs:

ubuntu-osd1:~$ ceph -s
2011-03-22 21:07:00.516925 pg v3103: 1056 pgs: 1056 active+clean; 1001 MB data, 4940 MB used, 182 GB / 196 GB avail
2011-03-22 21:07:00.518335 mds e11: 1/1/1 up {0=up:active}, 1 up:standby
2011-03-22 21:07:00.518360 osd e44: 4 osds: 4 up, 4 in
2011-03-22 21:07:00.518398 log 2011-03-22 00:45:35.004304 osd3 172.16.10.66:6803/6152 89 : [INF] 3.1p3 scrub ok
2011-03-22 21:07:00.518458 class rbd (v1.3 [x86-64])
2011-03-22 21:07:00.518472 mon e1: 3 mons at {0=172.16.10.171:6789/0,1=172.16.10.171:6790/0,2=172.16.10.171:6791/0}

  1. tail -f /var/log/messages
    Mar 22 10:46:56 ubuntu-client kernel: [ 948.758323] libceph: tid 115358 timed out on osd0, will reset osd
    Mar 22 10:46:56 ubuntu-client kernel: [ 948.767389] libceph: tid 111238 timed out on osd3, will reset osd
    Mar 22 10:47:56 ubuntu-client kernel: [ 1008.890012] libceph: tid 128526 timed out on osd3, will reset osd
    Mar 22 10:47:56 ubuntu-client kernel: [ 1008.898780] libceph: tid 128841 timed out on osd0, will reset osd
    Mar 22 10:47:56 ubuntu-client kernel: [ 1008.907657] libceph: tid 128637 timed out on osd1, will reset osd
    Mar 22 10:47:56 ubuntu-client kernel: [ 1008.916531] libceph: tid 128929 timed out on osd2, will reset osd
    Mar 22 10:48:56 ubuntu-client kernel: [ 1069.040017] libceph: tid 125017 timed out on osd2, will reset osd
    Mar 22 10:48:56 ubuntu-client kernel: [ 1069.050206] libceph: tid 123203 timed out on osd1, will reset osd
    Mar 22 10:48:56 ubuntu-client kernel: [ 1069.060196] libceph: tid 109631 timed out on osd0, will reset osd
    Mar 22 10:48:56 ubuntu-client kernel: [ 1069.070018] libceph: tid 101673 timed out on osd3, will reset osd
    Mar 22 10:49:56 ubuntu-client kernel: [ 1129.190013] libceph: tid 143305 timed out on osd3, will reset osd
    Mar 22 10:49:56 ubuntu-client kernel: [ 1129.200096] libceph: tid 143505 timed out on osd0, will reset osd
    Mar 22 10:49:56 ubuntu-client kernel: [ 1129.209906] libceph: tid 142874 timed out on osd1, will reset osd
    Mar 22 10:49:56 ubuntu-client kernel: [ 1129.219750] libceph: tid 143543 timed out on osd2, will reset osd
    Mar 22 10:50:57 ubuntu-client kernel: [ 1189.340010] libceph: tid 139507 timed out on osd2, will reset osd
    Mar 22 10:50:57 ubuntu-client kernel: [ 1189.350361] libceph: tid 137335 timed out on osd1, will reset osd
    Mar 22 10:50:57 ubuntu-client kernel: [ 1189.360568] libceph: tid 137986 timed out on osd3, will reset osd
    Mar 22 10:50:57 ubuntu-client kernel: [ 1189.370585] libceph: tid 135340 timed out on osd0, will reset osd
    Mar 22 10:51:57 ubuntu-client kernel: [ 1249.500010] libceph: tid 153039 timed out on osd0, will reset osd
    Mar 22 10:51:57 ubuntu-client kernel: [ 1249.509849] libceph: tid 152511 timed out on osd3, will reset osd
    Mar 22 10:51:57 ubuntu-client kernel: [ 1249.519782] libceph: tid 152689 timed out on osd1, will reset osd
    Mar 22 10:51:57 ubuntu-client kernel: [ 1249.529640] libceph: tid 153132 timed out on osd2, will reset osd
    Mar 22 10:52:57 ubuntu-client kernel: [ 1309.650012] libceph: tid 124807 timed out on osd2, will reset osd
    Mar 22 10:52:57 ubuntu-client kernel: [ 1309.660397] libceph: tid 148584 timed out on osd1, will reset osd
    Mar 22 10:52:57 ubuntu-client kernel: [ 1309.670513] libceph: tid 147318 timed out on osd3, will reset osd
    Mar 22 10:52:57 ubuntu-client kernel: [ 1309.680504] libceph: tid 144286 timed out on osd0, will reset osd
    Mar 22 10:53:57 ubuntu-client kernel: [ 1369.810011] libceph: tid 165429 timed out on osd0, will reset osd
    Mar 22 10:53:57 ubuntu-client kernel: [ 1369.819820] libceph: tid 165246 timed out on osd3, will reset osd
    Mar 22 10:53:57 ubuntu-client kernel: [ 1369.829895] libceph: tid 165310 timed out on osd1, will reset osd
    Mar 22 10:53:57 ubuntu-client kernel: [ 1369.839895] libceph: tid 165454 timed out on osd2, will reset osd
    Mar 22 10:54:57 ubuntu-client kernel: [ 1429.960012] libceph: tid 163748 timed out on osd2, will reset osd
    Mar 22 10:54:57 ubuntu-client kernel: [ 1429.970315] libceph: tid 161754 timed out on osd1, will reset osd
    Mar 22 10:54:57 ubuntu-client kernel: [ 1429.980509] libceph: tid 162641 timed out on osd3, will reset osd
    Mar 22 10:54:57 ubuntu-client kernel: [ 1429.990558] libceph: tid 156104 timed out on osd0, will reset osd
    Mar 22 10:55:57 ubuntu-client kernel: [ 1490.120012] libceph: tid 172157 timed out on osd0, will reset osd
    Mar 22 10:55:57 ubuntu-client kernel: [ 1490.129949] libceph: tid 171732 timed out on osd3, will reset osd
    Mar 22 10:55:57 ubuntu-client kernel: [ 1490.139926] libceph: tid 171855 timed out on osd1, will reset osd
    Mar 22 10:55:57 ubuntu-client kernel: [ 1490.149810] libceph: tid 172228 timed out on osd2, will reset osd
    Mar 22 10:56:57 ubuntu-client kernel: [ 1550.270012] libceph: tid 167512 timed out on osd2, will reset osd
    Mar 22 10:56:57 ubuntu-client kernel: [ 1550.279539] libceph: tid 160619 timed out on osd1, will reset osd
    Mar 22 10:56:57 ubuntu-client kernel: [ 1550.289065] libceph: tid 167756 timed out on osd3, will reset osd
    Mar 22 10:57:02 ubuntu-client kernel: [ 1555.300014] libceph: tid 172545 timed out on osd0, will reset osd
    Mar 22 10:57:58 ubuntu-client kernel: [ 1610.420012] libceph: tid 175697 timed out on osd3, will reset osd
    Mar 22 10:57:58 ubuntu-client kernel: [ 1610.428973] libceph: tid 175750 timed out on osd1, will reset osd
    Mar 22 10:57:58 ubuntu-client kernel: [ 1610.438061] libceph: tid 175335 timed out on osd2, will reset osd
    Mar 22 10:57:58 ubuntu-client kernel: [ 1611.011252] libceph: skipping osd2 172.16.10.67:6803 seq 1 expected 2
    Mar 22 10:58:58 ubuntu-client kernel: [ 1670.560011] libceph: tid 175335 timed out on osd2, will reset osd
    Mar 22 10:58:58 ubuntu-client kernel: [ 1670.568524] libceph: tid 154514 timed out on osd1, will reset osd
    Mar 22 10:58:58 ubuntu-client kernel: [ 1670.577133] libceph: tid 155310 timed out on osd3, will reset osd
    Mar 22 10:59:58 ubuntu-client kernel: [ 1730.700013] libceph: tid 134814 timed out on osd3, will reset osd
    Mar 22 10:59:58 ubuntu-client kernel: [ 1730.707931] libceph: tid 145838 timed out on osd1, will reset osd
    Mar 22 10:59:58 ubuntu-client kernel: [ 1730.716044] libceph: tid 107651 timed out on osd2, will reset osd
    Mar 22 11:00:58 ubuntu-client kernel: [ 1790.840013] libceph: tid 105967 timed out on osd2, will reset osd
    Mar 22 11:00:58 ubuntu-client kernel: [ 1790.847502] libceph: tid 109902 timed out on osd1, will reset osd
    Mar 22 11:00:58 ubuntu-client kernel: [ 1790.855090] libceph: tid 97628 timed out on osd3, will reset osd
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.685669] iozone_x86_64 D ffff880076524958 0 1057 1052 0x00000000
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.693141] ffff880076447cd8 0000000000000086 ffff880076447c38 ffffffff00000000
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.700991] 0000000000013c40 ffff8800765245c0 ffff880076524958 ffff880076447fd8
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.708830] ffff880076524960 0000000000013c40 ffff880076446010 0000000000013c40
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.716694] Call Trace:
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.719582] [<ffffffff810f8e40>] ? sync_page+0x0/0x60
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.725192] [<ffffffff8158b204>] io_schedule+0x44/0x60
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.730867] [<ffffffff810f8e85>] sync_page+0x45/0x60
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.736374] [<ffffffff8158b83f>] __wait_on_bit+0x5f/0x90
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.742237] [<ffffffff810f9053>] wait_on_page_bit+0x73/0x80
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.748360] [<ffffffff81082a20>] ? wake_bit_function+0x0/0x40
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.754677] [<ffffffff81104135>] ? pagevec_lookup_tag+0x25/0x40
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.761169] [<ffffffff810f94a3>] filemap_fdatawait_range+0x113/0x1c0
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.768090] [<ffffffff810f9650>] filemap_write_and_wait_range+0x70/0x80
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.775292] [<ffffffff8117a4ba>] vfs_fsync_range+0x5a/0x90
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.781370] [<ffffffff8117a55c>] vfs_fsync+0x1c/0x20
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.786924] [<ffffffff8117a59a>] do_fsync+0x3a/0x60
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.792410] [<ffffffff8117a5f0>] sys_fsync+0x10/0x20
    Mar 22 11:01:13 ubuntu-client kernel: [ 1805.797971] [<ffffffff8100bfc2>] system_call_fastpath+0x16/0x1b
    Mar 22 11:01:58 ubuntu-client kernel: [ 1850.980018] libceph: tid 102000 timed out on osd1, will reset osd
    Mar 22 11:01:58 ubuntu-client kernel: [ 1850.987255] libceph: tid 101451 timed out on osd2, will reset osd
    Mar 22 11:02:58 ubuntu-client kernel: [ 1911.110011] libceph: tid 90126 timed out on osd2, will reset osd
    Mar 22 11:02:58 ubuntu-client kernel: [ 1911.116834] libceph: tid 96191 timed out on osd1, will reset osd
    Mar 22 11:03:58 ubuntu-client kernel: [ 1971.240011] libceph: tid 91928 timed out on osd1, will reset osd
    Mar 22 11:06:12 ubuntu-client kernel: [ 2105.160013] libceph: tid 221591 timed out on osd3, will reset osd
    Mar 22 11:06:12 ubuntu-client kernel: [ 2105.168286] libceph: tid 222121 timed out on osd1, will reset osd
    Mar 22 11:06:12 ubuntu-client kernel: [ 2105.176594] libceph: tid 223623 timed out on osd0, will reset osd
    Mar 22 11:06:12 ubuntu-client kernel: [ 2105.184786] libceph: tid 225922 timed out on osd2, will reset osd
    Mar 22 11:07:12 ubuntu-client kernel: [ 2165.310011] libceph: tid 239673 timed out on osd2, will reset osd
    Mar 22 11:07:12 ubuntu-client kernel: [ 2165.318553] libceph: tid 239682 timed out on osd0, will reset osd
    Mar 22 11:07:12 ubuntu-client kernel: [ 2165.327226] libceph: tid 239691 timed out on osd1, will reset osd
    Mar 22 11:07:13 ubuntu-client kernel: [ 2165.335742] libceph: tid 239690 timed out on osd3, will reset osd
    Mar 22 11:08:13 ubuntu-client kernel: [ 2225.460011] libceph: tid 245949 timed out on osd3, will reset osd
    Mar 22 11:08:13 ubuntu-client kernel: [ 2225.468123] libceph: tid 245569 timed out on osd1, will reset osd
    Mar 22 11:08:13 ubuntu-client kernel: [ 2225.476265] libceph: tid 245630 timed out on osd0, will reset osd
    Mar 22 11:08:13 ubuntu-client kernel: [ 2225.484338] libceph: tid 245911 timed out on osd2, will reset osd
    Mar 22 11:09:13 ubuntu-client kernel: [ 2285.610011] libceph: tid 247438 timed out on osd2, will reset osd
    Mar 22 11:09:13 ubuntu-client kernel: [ 2285.618322] libceph: tid 247342 timed out on osd0, will reset osd
    Mar 22 11:09:13 ubuntu-client kernel: [ 2285.626759] libceph: tid 247444 timed out on osd3, will reset osd
    Mar 22 11:09:23 ubuntu-client kernel: [ 2295.650014] libceph: tid 247445 timed out on osd1, will reset osd
    Mar 22 11:10:13 ubuntu-client kernel: [ 2345.750013] libceph: tid 258786 timed out on osd3, will reset osd
    Mar 22 11:10:13 ubuntu-client kernel: [ 2345.758134] libceph: tid 257932 timed out on osd0, will reset osd
    Mar 22 11:10:13 ubuntu-client kernel: [ 2345.766403] libceph: tid 258682 timed out on osd2, will reset osd
    Mar 22 11:11:13 ubuntu-client kernel: [ 2405.890010] libceph: tid 253515 timed out on osd2, will reset osd
    Mar 22 11:11:13 ubuntu-client kernel: [ 2405.897092] libceph: tid 250390 timed out on osd0, will reset osd
    Mar 22 11:11:13 ubuntu-client kernel: [ 2405.904264] libceph: tid 253241 timed out on osd3, will reset osd
    Mar 22 11:12:13 ubuntu-client kernel: [ 2466.030011] libceph: tid 236134 timed out on osd0, will reset osd
    Mar 22 11:15:08 ubuntu-client kernel: [ 2640.610012] libceph: tid 283943 timed out on osd0, will reset osd
    Mar 22 11:15:08 ubuntu-client kernel: [ 2640.617736] libceph: tid 284038 timed out on osd2, will reset osd
    Mar 22 11:15:08 ubuntu-client kernel: [ 2640.625531] libceph: tid 285011 timed out on osd3, will reset osd
    Mar 22 11:15:08 ubuntu-client kernel: [ 2640.633223] libceph: tid 285058 timed out on osd1, will reset osd
    Mar 22 11:16:08 ubuntu-client kernel: [ 2700.760012] libceph: tid 298116 timed out on osd1, will reset osd
    Mar 22 11:16:08 ubuntu-client kernel: [ 2700.767674] libceph: tid 297734 timed out on osd3, will reset osd
    Mar 22 11:16:08 ubuntu-client kernel: [ 2700.775455] libceph: tid 297886 timed out on osd2, will reset osd
    Mar 22 11:16:08 ubuntu-client kernel: [ 2700.783184] libceph: tid 298168 timed out on osd0, will reset osd
    Mar 22 11:17:08 ubuntu-client kernel: [ 2760.910011] libceph: tid 301530 timed out on osd0, will reset osd
    Mar 22 11:17:08 ubuntu-client kernel: [ 2760.916833] libceph: tid 301462 timed out on osd2, will reset osd
    Mar 22 11:17:08 ubuntu-client kernel: [ 2760.923551] libceph: tid 301436 timed out on osd3, will reset osd
    Mar 22 11:19:45 ubuntu-client kernel: [ 2917.590015] libceph: tid 318599 timed out on osd2, will reset osd
    Mar 22 11:36:22 ubuntu-client kernel: [ 3915.230011] libceph: tid 539580 timed out on osd3, will reset osd
    Mar 22 11:36:22 ubuntu-client kernel: [ 3915.241271] libceph: tid 541777 timed out on osd2, will reset osd
    Mar 22 11:36:22 ubuntu-client kernel: [ 3915.252391] libceph: tid 542059 timed out on osd1, will reset osd
    Mar 22 11:36:22 ubuntu-client kernel: [ 3915.263542] libceph: tid 542445 timed out on osd0, will reset osd
    Mar 22 11:37:23 ubuntu-client kernel: [ 3975.390014] libceph: tid 595281 timed out on osd0, will reset osd
    Mar 22 11:37:23 ubuntu-client kernel: [ 3975.401294] libceph: tid 595274 timed out on osd1, will reset osd
    Mar 22 11:37:23 ubuntu-client kernel: [ 3975.412529] libceph: tid 595280 timed out on osd2, will reset osd
    Mar 22 11:37:23 ubuntu-client kernel: [ 3975.423621] libceph: tid 595194 timed out on osd3, will reset osd
    Mar 22 11:38:23 ubuntu-client kernel: [ 4035.550011] libceph: tid 586690 timed out on osd3, will reset osd
    Mar 22 11:38:23 ubuntu-client kernel: [ 4035.560594] libceph: tid 591425 timed out on osd2, will reset osd
    Mar 22 11:38:23 ubuntu-client kernel: [ 4035.571146] libceph: tid 582250 timed out on osd1, will reset osd
    Mar 22 11:38:23 ubuntu-client kernel: [ 4035.581742] libceph: tid 592116 timed out on osd0, will reset osd
    Mar 22 11:39:23 ubuntu-client kernel: [ 4095.710012] libceph: tid 586787 timed out on osd0, will reset osd
    Mar 22 11:39:23 ubuntu-client kernel: [ 4095.720957] libceph: tid 581408 timed out on osd1, will reset osd
    Mar 22 11:39:23 ubuntu-client kernel: [ 4095.731729] libceph: tid 584244 timed out on osd2, will reset osd
    Mar 22 11:39:23 ubuntu-client kernel: [ 4095.742416] libceph: tid 583587 timed out on osd3, will reset osd
    Mar 22 11:40:23 ubuntu-client kernel: [ 4155.870013] libceph: tid 602053 timed out on osd3, will reset osd
    Mar 22 11:40:23 ubuntu-client kernel: [ 4155.880454] libceph: tid 602097 timed out on osd2, will reset osd
    Mar 22 11:40:23 ubuntu-client kernel: [ 4155.890709] libceph: tid 602174 timed out on osd1, will reset osd
    Mar 22 11:40:23 ubuntu-client kernel: [ 4155.900978] libceph: tid 602204 timed out on osd0, will reset osd
    Mar 22 11:41:23 ubuntu-client kernel: [ 4216.030011] libceph: tid 596132 timed out on osd0, will reset osd
    Mar 22 11:41:23 ubuntu-client kernel: [ 4216.041005] libceph: tid 570851 timed out on osd1, will reset osd
    Mar 22 11:41:23 ubuntu-client kernel: [ 4216.051747] libceph: tid 573826 timed out on osd2, will reset osd
    .......................................................................................................
    Mar 22 12:02:27 ubuntu-client kernel: [ 5479.390947] libceph: tid 740652 timed out on osd1, will reset osd
    Mar 22 12:02:27 ubuntu-client kernel: [ 5479.401794] libceph: tid 733253 timed out on osd0, will reset osd
    Mar 22 12:03:02 ubuntu-client kernel: [ 5514.480013] libceph: tid 747953 timed out on osd3, will reset osd
    Mar 22 12:03:27 ubuntu-client kernel: [ 5539.540019] libceph: tid 748153 timed out on osd0, will reset osd
    Mar 22 12:03:27 ubuntu-client kernel: [ 5539.553914] libceph: tid 748121 timed out on osd1, will reset osd
    Mar 22 12:03:27 ubuntu-client kernel: [ 5539.564888] libceph: tid 748025 timed out on osd2, will reset osd
    Mar 22 12:04:27 ubuntu-client kernel: [ 5599.690011] libceph: tid 749173 timed out on osd3, will reset osd
    Mar 22 12:04:27 ubuntu-client kernel: [ 5599.700632] libceph: tid 752137 timed out on osd2, will reset osd
    Mar 22 12:04:27 ubuntu-client kernel: [ 5599.711256] libceph: tid 752185 timed out on osd1, will reset osd
    Mar 22 12:04:27 ubuntu-client kernel: [ 5599.721748] libceph: tid 752199 timed out on osd0, will reset osd
    Mar 22 12:05:27 ubuntu-client kernel: [ 5659.850014] libceph: tid 753220 timed out on osd0, will reset osd
    Mar 22 12:05:27 ubuntu-client kernel: [ 5659.863625] libceph: tid 753198 timed out on osd2, will reset osd
    Mar 22 12:05:37 ubuntu-client kernel: [ 5669.890012] libceph: tid 753224 timed out on osd3, will reset osd
    Mar 22 12:05:37 ubuntu-client kernel: [ 5669.900697] libceph: tid 753619 timed out on osd1, will reset osd
    Mar 22 12:06:27 ubuntu-client kernel: [ 5720.010022] libceph: tid 764305 timed out on osd2, will reset osd
    Mar 22 12:06:27 ubuntu-client kernel: [ 5720.020362] libceph: tid 764307 timed out on osd0, will reset osd
    Mar 22 12:07:27 ubuntu-client kernel: [ 5780.150013] libceph: tid 762955 timed out on osd0, will reset osd
    Mar 22 12:07:27 ubuntu-client kernel: [ 5780.162627] libceph: tid 755339 timed out on osd2, will reset osd
    Mar 22 12:08:27 ubuntu-client kernel: [ 5840.290018] libceph: tid 759879 timed out on osd0, will reset osd
    Mar 22 12:09:28 ubuntu-client kernel: [ 5900.420012] libceph: tid 756522 timed out on osd0, will reset osd
    Mar 22 12:10:28 ubuntu-client kernel: [ 5960.550012] libceph: tid 749745 timed out on osd0, will reset osd
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.815595] iozone_x86_64 D ffff880076524958 0 1057 1052 0x00000000
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.823045] ffff880076447cd8 0000000000000086 ffff880076447c38 ffffffff00000000
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.830861] 0000000000013c40 ffff8800765245c0 ffff880076524958 ffff880076447fd8
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.838678] ffff880076524960 0000000000013c40 ffff880076446010 0000000000013c40
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.846519] Call Trace:
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.849382] [<ffffffff810f8e40>] ? sync_page+0x0/0x60
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.854956] [<ffffffff8158b204>] io_schedule+0x44/0x60
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.860615] [<ffffffff810f8e85>] sync_page+0x45/0x60
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.866094] [<ffffffff8158b83f>] __wait_on_bit+0x5f/0x90
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.871948] [<ffffffff810f9053>] wait_on_page_bit+0x73/0x80
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.878052] [<ffffffff81082a20>] ? wake_bit_function+0x0/0x40
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.884344] [<ffffffff81104135>] ? pagevec_lookup_tag+0x25/0x40
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.890811] [<ffffffff810f94a3>] filemap_fdatawait_range+0x113/0x1c0
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.897710] [<ffffffff810f9650>] filemap_write_and_wait_range+0x70/0x80
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.904901] [<ffffffff8117a4ba>] vfs_fsync_range+0x5a/0x90
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.910960] [<ffffffff8117a55c>] vfs_fsync+0x1c/0x20
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.916490] [<ffffffff8117a59a>] do_fsync+0x3a/0x60
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.921951] [<ffffffff8117a5f0>] sys_fsync+0x10/0x20
    Mar 22 12:11:13 ubuntu-client kernel: [ 6005.927488] [<ffffffff8100bfc2>] system_call_fastpath+0x16/0x1b
    Mar 22 12:11:28 ubuntu-client kernel: [ 6020.680011] libceph: tid 720380 timed out on osd0, will reset osd
    Mar 22 12:12:28 ubuntu-client kernel: [ 6080.810012] libceph: tid 711512 timed out on osd0, will reset osd

ceph.conf View (3.18 KB) changping Wu, 03/21/2011 10:35 PM

History

#1 Updated by Sage Weil almost 13 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF