Project

General

Profile

Actions

Bug #3655

closed

client: hang in fsstress

Added by Sam Lang over 11 years ago. Updated almost 8 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

fsstress stuck in _read_sync()

#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x000000000073f1f8 in Cond::Wait (this=0x7fff229635c0, mutex=...) at ./common/Cond.h:55
#2 0x00000000007299bd in Client::_read_sync (this=0x1ed2800, f=0x1ebce10, off=0, len=131072,
bl=0x7fff229639a0) at client/Client.cc:5570
#3 0x00000000007284b0 in Client::_read (this=0x1ed2800, f=0x1ebce10, offset=0, size=131072,
bl=0x7fff229639a0) at client/Client.cc:5429
#4 0x0000000000739d83 in Client::ll_read (this=0x1ed2800, fh=0x1ebce10, off=0, len=131072,
bl=0x7fff229639a0) at client/Client.cc:7151
#5 0x00000000006ee8c3 in ceph_ll_read (req=0x1ec8000, ino=1099511627957, size=131072, off=0,
fi=0x7fff22963a20) at client/fuse_ll.cc:340
#6 0x00007f5f781a3d5e in ?? () from /lib/libfuse.so.2
#7 0x00007f5f781a1d65 in fuse_session_loop () from /lib/libfuse.so.2
#8 0x00000000006ef8d5 in ceph_fuse_ll_main (c=0x1ed2800, argc=2, argv=0x1eb8050, fd=0)
at client/fuse_ll.cc:640
#9 0x00000000006ec602 in main (argc=7, argv=0x7fff22964498, envp=0x1eba000) at ceph_fuse.cc:157
(gdb)

The output of fsstress is:

2012-12-19T09:02:56.379 INFO:teuthology.task.workunit.client.0.out:Starting fsstress /usr/lib/ltp/testcases/bin/fsstress -d fsstress-plana551846 -l 1 -n 1000 -p 10 -v
2012-12-19T09:02:56.441 INFO:teuthology.task.workunit.client.0.out:seed = 1356575221
2012-12-19T09:02:56.452 INFO:teuthology.task.workunit.client.0.out:0/0: dread - no filename
2012-12-19T09:02:56.452 INFO:teuthology.task.workunit.client.0.out:5/0: truncate - no filename
2012-12-19T09:02:56.453 INFO:teuthology.task.workunit.client.0.out:5/1: chown . 3 1
2012-12-19T09:02:56.454 INFO:teuthology.task.workunit.client.0.out:5/2: readlink - no filename
2012-12-19T09:02:56.454 INFO:teuthology.task.workunit.client.0.out:5/3: truncate - no filename
2012-12-19T09:02:56.454 INFO:teuthology.task.workunit.client.0.out:1/0: truncate - no filename
2012-12-19T09:02:56.454 INFO:teuthology.task.workunit.client.0.out:1/1: write - no filename
2012-12-19T09:02:56.454 INFO:teuthology.task.workunit.client.0.out:1/2: link - no file
2012-12-19T09:02:56.455 INFO:teuthology.task.workunit.client.0.out:1/3: dwrite - no filename
2012-12-19T09:02:56.455 INFO:teuthology.task.workunit.client.0.out:1/4: dread - no filename
2012-12-19T09:02:56.457 INFO:teuthology.task.workunit.client.0.out:0/1: getdents . 0
2012-12-19T09:02:56.457 INFO:teuthology.task.workunit.client.0.out:0/2: write - no filename
2012-12-19T09:02:56.457 INFO:teuthology.task.workunit.client.0.out:0/3: write - no filename
2012-12-19T09:02:56.457 INFO:teuthology.task.workunit.client.0.out:0/4: stat - no entries
2012-12-19T09:02:56.458 INFO:teuthology.task.workunit.client.0.out:0/5: fsync - no filename
2012-12-19T09:02:56.458 INFO:teuthology.task.workunit.client.0.out:0/6: write - no filename
2012-12-19T09:02:56.458 INFO:teuthology.task.workunit.client.0.out:2/0: truncate - no filename
2012-12-19T09:02:56.458 INFO:teuthology.task.workunit.client.0.out:2/1: dwrite - no filename
2012-12-19T09:02:56.458 INFO:teuthology.task.workunit.client.0.out:2/2: truncate - no filename
2012-12-19T09:02:56.459 INFO:teuthology.task.workunit.client.0.out:2/3: readlink - no filename
2012-12-19T09:02:56.459 INFO:teuthology.task.workunit.client.0.out:2/4: link - no file
2012-12-19T09:02:56.459 INFO:teuthology.task.workunit.client.0.out:2/5: dread - no filename
2012-12-19T09:02:56.459 INFO:teuthology.task.workunit.client.0.out:2/6: dread - no filename
2012-12-19T09:02:56.459 INFO:teuthology.task.workunit.client.0.out:2/7: dread - no filename
2012-12-19T09:02:56.460 INFO:teuthology.task.workunit.client.0.out:1/5: mkdir d0 0
2012-12-19T09:02:56.460 INFO:teuthology.task.workunit.client.0.out:1/6: write - no filename
2012-12-19T09:02:56.463 INFO:teuthology.task.workunit.client.0.out:0/7: symlink l0 0
2012-12-19T09:02:56.463 INFO:teuthology.task.workunit.client.0.out:0/8: dwrite - no filename
2012-12-19T09:02:56.469 INFO:teuthology.task.workunit.client.0.out:4/0: fdatasync - no filename
2012-12-19T09:02:56.469 INFO:teuthology.task.workunit.client.0.out:4/1: rename - no filename
2012-12-19T09:02:56.470 INFO:teuthology.task.workunit.client.0.out:4/2: stat - no entries
2012-12-19T09:02:56.470 INFO:teuthology.task.workunit.client.0.out:4/3: dread - no filename
2012-12-19T09:02:56.470 INFO:teuthology.task.workunit.client.0.out:4/4: stat - no entries
2012-12-19T09:02:56.470 INFO:teuthology.task.workunit.client.0.out:0/9: symlink l1 0
2012-12-19T09:02:56.474 INFO:teuthology.task.workunit.client.0.out:3/0: mkdir d0 0
2012-12-19T09:02:56.474 INFO:teuthology.task.workunit.client.0.out:3/1: dread - no filename
2012-12-19T09:02:56.474 INFO:teuthology.task.workunit.client.0.out:3/2: write - no filename
2012-12-19T09:02:56.476 INFO:teuthology.task.workunit.client.0.out:1/7: creat d0/f1 x:0 0 0
2012-12-19T09:02:56.476 INFO:teuthology.task.workunit.client.0.out:4/5: mknod c0 1
2012-12-19T09:02:56.476 INFO:teuthology.task.workunit.client.0.out:6/0: read - no filename
2012-12-19T09:02:56.477 INFO:teuthology.task.workunit.client.0.out:6/1: rename - no filename
2012-12-19T09:02:56.479 INFO:teuthology.task.workunit.client.0.out:0/10: mkdir d2 0
2012-12-19T09:02:56.479 INFO:teuthology.task.workunit.client.0.out:6/2: chown . 2532756 1
2012-12-19T09:02:56.480 INFO:teuthology.task.workunit.client.0.out:6/3: chown . 13 1
2012-12-19T09:02:56.480 INFO:teuthology.task.workunit.client.0.out:6/4: write - no filename
2012-12-19T09:02:56.480 INFO:teuthology.task.workunit.client.0.out:6/5: write - no filename
2012-12-19T09:02:56.480 INFO:teuthology.task.workunit.client.0.out:6/6: link - no file
2012-12-19T09:02:56.480 INFO:teuthology.task.workunit.client.0.out:6/7: truncate - no filename
2012-12-19T09:02:56.481 INFO:teuthology.task.workunit.client.0.out:3/3: getdents d0 0
2012-12-19T09:02:56.481 INFO:teuthology.task.workunit.client.0.out:3/4: read - no filename
2012-12-19T09:02:56.481 INFO:teuthology.task.workunit.client.0.out:3/5: readlink - no filename
2012-12-19T09:02:56.482 INFO:teuthology.task.workunit.client.0.out:3/6: write - no filename
2012-12-19T09:02:56.482 INFO:teuthology.task.workunit.client.0.out:8/0: link - no file
2012-12-19T09:02:56.490 INFO:teuthology.task.workunit.client.0.out:8/1: chown . 7122355 1
2012-12-19T09:02:56.490 INFO:teuthology.task.workunit.client.0.out:6/8: creat f0 x:0 0 0
2012-12-19T09:02:56.491 INFO:teuthology.task.workunit.client.0.out:7/0: creat f0 x:0 0 0
2012-12-19T09:02:56.491 INFO:teuthology.task.workunit.client.0.out:9/0: dwrite - no filename
2012-12-19T09:02:56.493 INFO:teuthology.task.workunit.client.0.out:3/7: symlink d0/l1 0
2012-12-19T09:02:56.493 INFO:teuthology.task.workunit.client.0.out:3/8: truncate - no filename
2012-12-19T09:02:56.493 INFO:teuthology.task.workunit.client.0.out:3/9: dwrite - no filename
2012-12-19T09:02:56.494 INFO:teuthology.task.workunit.client.0.out:3/10: fdatasync - no filename
2012-12-19T09:02:56.494 INFO:teuthology.task.workunit.client.0.out:8/2: mknod c0 1
2012-12-19T09:02:56.494 INFO:teuthology.task.workunit.client.0.out:8/3: link - no file
2012-12-19T09:02:56.560 INFO:teuthology.task.workunit.client.0.out:1/8: truncate d0/f1 8951 0
2012-12-19T09:02:56.560 INFO:teuthology.task.workunit.client.0.out:7/1: stat f0 0
2012-12-19T09:02:56.569 INFO:teuthology.task.workunit.client.0.out:6/9: creat f1 x:0 0 0
2012-12-19T09:02:56.572 INFO:teuthology.task.workunit.client.0.out:8/4: symlink l1 0
2012-12-19T09:02:56.572 INFO:teuthology.task.workunit.client.0.out:8/5: dread - no filename
2012-12-19T09:02:56.572 INFO:teuthology.task.workunit.client.0.out:8/6: fdatasync - no filename
2012-12-19T09:02:56.572 INFO:teuthology.task.workunit.client.0.out:9/1: creat f0 x:0 0 0
2012-12-19T09:02:56.573 INFO:teuthology.task.workunit.client.0.out:3/11: symlink d0/l2 0
2012-12-19T09:02:56.574 INFO:teuthology.task.workunit.client.0.out:3/12: dread - no filename
2012-12-19T09:02:56.580 INFO:teuthology.task.workunit.client.0.out:1/9: creat d0/f2 x:0 0 0
2012-12-19T09:02:56.590 INFO:teuthology.task.workunit.client.0.out:3/13: stat d0/l2 0
2012-12-19T09:02:56.590 INFO:teuthology.task.workunit.client.0.out:8/7: creat f2 x:0 0 0
2012-12-19T09:02:56.590 INFO:teuthology.task.workunit.client.0.out:3/14: dread - no filename
2012-12-19T09:02:56.590 INFO:teuthology.task.workunit.client.0.out:3/15: dwrite - no filename
2012-12-19T09:02:56.590 INFO:teuthology.task.workunit.client.0.out:3/16: write - no filename
2012-12-19T09:02:56.591 INFO:teuthology.task.workunit.client.0.out:3/17: read - no filename
2012-12-19T09:02:56.591 INFO:teuthology.task.workunit.client.0.out:3/18: truncate - no filename
2012-12-19T09:02:56.591 INFO:teuthology.task.workunit.client.0.out:3/19: write - no filename
2012-12-19T09:02:56.591 INFO:teuthology.task.workunit.client.0.out:3/20: truncate - no filename
2012-12-19T09:02:56.591 INFO:teuthology.task.workunit.client.0.out:3/21: read - no filename
2012-12-19T09:02:56.593 INFO:teuthology.task.workunit.client.0.out:6/10: rename f0 to f2 0
2012-12-19T09:02:56.595 INFO:teuthology.task.workunit.client.0.out:3/22: stat d0/l2 0
2012-12-19T09:02:56.595 INFO:teuthology.task.workunit.client.0.out:3/23: dread - no filename
2012-12-19T09:02:56.597 INFO:teuthology.task.workunit.client.0.out:6/11: write f1 [180304,42579] 0
2012-12-19T09:02:56.654 INFO:teuthology.task.workunit.client.0.out:5/4: sync
2012-12-19T09:02:56.654 INFO:teuthology.task.workunit.client.0.out:4/6: sync
2012-12-19T09:02:56.654 INFO:teuthology.task.workunit.client.0.out:0/11: sync
2012-12-19T09:02:56.655 INFO:teuthology.task.workunit.client.0.out:2/8: sync
2012-12-19T09:02:56.657 INFO:teuthology.task.workunit.client.0.out:5/5: sync
2012-12-19T09:02:56.792 INFO:teuthology.task.workunit.client.0.out:1/10: link d0/f1 d0/f3 0
2012-12-19T09:02:56.792 INFO:teuthology.task.workunit.client.0.out:3/24: readlink d0/l2 0
2012-12-19T09:02:56.845 INFO:teuthology.task.workunit.client.0.out:2/9: mknod c0 1
2012-12-19T09:02:56.845 INFO:teuthology.task.workunit.client.0.out:2/10: fsync - no filename
2012-12-19T09:02:56.845 INFO:teuthology.task.workunit.client.0.out:2/11: write - no filename
2012-12-19T09:02:56.845 INFO:teuthology.task.workunit.client.0.out:2/12: write - no filename
2012-12-19T09:02:56.845 INFO:teuthology.task.workunit.client.0.out:5/6: mknod c0 1
2012-12-19T09:02:56.848 INFO:teuthology.task.workunit.client.0.out:2/13: chown . 24156706 1
2012-12-19T09:02:56.849 INFO:teuthology.task.workunit.client.0.out:2/14: write - no filename
2012-12-19T09:02:56.849 INFO:teuthology.task.workunit.client.0.out:2/15: dread - no filename
2012-12-19T09:02:56.849 INFO:teuthology.task.workunit.client.0.out:2/16: write - no filename
2012-12-19T09:02:56.850 INFO:teuthology.task.workunit.client.0.out:3/25: rmdir d0 39
2012-12-19T09:02:56.850 INFO:teuthology.task.workunit.client.0.out:3/26: truncate - no filename
2012-12-19T09:02:56.850 INFO:teuthology.task.workunit.client.0.out:3/27: dread - no filename
2012-12-19T09:02:56.851 INFO:teuthology.task.workunit.client.0.out:0/12: mknod d2/c3 1
2012-12-19T09:02:56.851 INFO:teuthology.task.workunit.client.0.out:0/13: read - no filename
2012-12-19T09:02:56.851 INFO:teuthology.task.workunit.client.0.out:0/14: dread - no filename
2012-12-19T09:02:56.851 INFO:teuthology.task.workunit.client.0.out:0/15: fsync - no filename
2012-12-19T09:02:56.852 INFO:teuthology.task.workunit.client.0.out:2/17: chown . 204144032 1
2012-12-19T09:02:56.852 INFO:teuthology.task.workunit.client.0.out:2/18: dread - no filename
2012-12-19T09:02:56.852 INFO:teuthology.task.workunit.client.0.out:2/19: dwrite - no filename
2012-12-19T09:02:56.853 INFO:teuthology.task.workunit.client.0.out:6/12: dread - f2 zero size
2012-12-19T09:02:56.853 INFO:teuthology.task.workunit.client.0.out:4/7: creat f1 x:0 0 0
2012-12-19T09:02:56.855 INFO:teuthology.task.workunit.client.0.out:2/20: chown . 306 1
2012-12-19T09:02:56.855 INFO:teuthology.task.workunit.client.0.out:2/21: read - no filename
2012-12-19T09:02:56.855 INFO:teuthology.task.workunit.client.0.out:2/22: write - no filename
2012-12-19T09:02:56.856 INFO:teuthology.task.workunit.client.0.out:2/23: write - no filename
2012-12-19T09:02:56.857 INFO:teuthology.task.workunit.client.0.out:2/24: truncate - no filename
2012-12-19T09:02:56.858 INFO:teuthology.task.workunit.client.0.out:2/25: stat - no entries
2012-12-19T09:02:56.859 INFO:teuthology.task.workunit.client.0.out:5/7: mkdir d1 0
2012-12-19T09:02:56.860 INFO:teuthology.task.workunit.client.0.out:5/8: dread - no filename
2012-12-19T09:02:56.860 INFO:teuthology.task.workunit.client.0.out:5/9: dread - no filename
2012-12-19T09:02:56.860 INFO:teuthology.task.workunit.client.0.out:2/26: chown . 28939 1
2012-12-19T09:02:56.860 INFO:teuthology.task.workunit.client.0.out:2/27: readlink - no filename
2012-12-19T09:02:56.863 INFO:teuthology.task.workunit.client.0.out:5/10: chown d1 1685839 1
2012-12-19T09:02:57.001 INFO:teuthology.task.workunit.client.0.out:1/11: write d0/f1 [310864,90078] 0
2012-12-19T09:02:57.003 INFO:teuthology.task.workunit.client.0.out:4/8: getdents . 0
2012-12-19T09:02:57.006 INFO:teuthology.task.workunit.client.0.out:0/16: symlink d2/l4 0
2012-12-19T09:02:57.008 INFO:teuthology.task.workunit.client.0.out:3/28: symlink d0/l3 0
2012-12-19T09:02:57.012 INFO:teuthology.task.workunit.client.0.out:2/28: symlink l1 0
2012-12-19T09:02:57.017 INFO:teuthology.task.workunit.client.0.out:3/29: rmdir d0 39
2012-12-19T09:02:57.019 INFO:teuthology.task.workunit.client.0.out:2/29: mkdir d2 0
2012-12-19T09:02:57.024 INFO:teuthology.task.workunit.client.0.out:4/9: getdents . 0
2012-12-19T09:02:57.025 INFO:teuthology.task.workunit.client.0.out:3/30: chown d0 16758662 1
2012-12-19T09:02:57.026 INFO:teuthology.task.workunit.client.0.out:5/11: creat d1/f2 x:0 0 0
2012-12-19T09:02:57.243 INFO:teuthology.task.workunit.client.0.out:0/17: link l1 d2/l5 0
2012-12-19T09:02:57.574 INFO:teuthology.task.workunit.client.0.out:1/12: link d0/f2 d0/f4 0
2012-12-19T09:02:57.575 INFO:teuthology.task.workunit.client.0.out:4/10: dread - f1 zero size
2012-12-19T09:02:57.577 INFO:teuthology.task.workunit.client.0.out:4/11: readlink - no filename
2012-12-19T09:02:57.577 INFO:teuthology.task.workunit.client.0.out:5/12: mknod d1/c3 1
2012-12-19T09:02:57.577 INFO:teuthology.task.workunit.client.0.out:7/2: dwrite f0 [0,4194304] 0
2012-12-19T09:02:57.889 INFO:teuthology.task.workunit.client.0.out:2/30: link l1 d2/l3 0
2012-12-19T09:02:57.889 INFO:teuthology.task.workunit.client.0.out:2/31: dwrite - no filename
2012-12-19T09:02:57.890 INFO:teuthology.task.workunit.client.0.out:0/18: rename d2 to d2/d6 22
2012-12-19T09:02:57.891 INFO:teuthology.task.workunit.client.0.out:3/31: creat d0/f4 x:0 0 0
2012-12-19T09:02:57.892 INFO:teuthology.task.workunit.client.0.out:2/32: chown d2 44 1
2012-12-19T09:02:57.892 INFO:teuthology.task.workunit.client.0.out:2/33: write - no filename
2012-12-19T09:02:57.892 INFO:teuthology.task.workunit.client.0.out:9/2: dwrite f0 [0,4194304] 0
2012-12-19T09:02:57.898 INFO:teuthology.task.workunit.client.0.out:5/13: getdents d1 0
2012-12-19T09:02:57.898 INFO:teuthology.task.workunit.client.0.out:1/13: creat d0/f5 x:0 0 0
2012-12-19T09:02:57.903 INFO:teuthology.task.workunit.client.0.out:4/12: rename f1 to f2 0
2012-12-19T09:02:57.904 INFO:teuthology.task.workunit.client.0.out:9/3: chown f0 0 1
2012-12-19T09:02:57.907 INFO:teuthology.task.workunit.client.0.out:3/32: mkdir d0/d5 0
2012-12-19T09:02:57.911 INFO:teuthology.task.workunit.client.0.out:2/34: symlink d2/l4 0
2012-12-19T09:02:57.912 INFO:teuthology.task.workunit.client.0.out:4/13: mknod c3 1
2012-12-19T09:02:57.914 INFO:teuthology.task.workunit.client.0.out:8/8: dwrite f2 [0,4194304] 0
2012-12-19T09:02:57.924 INFO:teuthology.task.workunit.client.0.out:0/19: rename d2/l5 to d2/l7 0
2012-12-19T09:02:57.925 INFO:teuthology.task.workunit.client.0.out:5/14: creat d1/f4 x:0 0 0
2012-12-19T09:02:57.925 INFO:teuthology.task.workunit.client.0.out:1/14: creat d0/f6 x:0 0 0
2012-12-19T09:02:57.926 INFO:teuthology.task.workunit.client.0.out:2/35: getdents d2 0
2012-12-19T09:02:57.926 INFO:teuthology.task.workunit.client.0.out:2/36: dread - no filename
2012-12-19T09:02:57.927 INFO:teuthology.task.workunit.client.0.out:2/37: write - no filename
2012-12-19T09:02:57.927 INFO:teuthology.task.workunit.client.0.out:9/4: write f0 [4926702,83486] 0
2012-12-19T09:02:57.928 INFO:teuthology.task.workunit.client.0.out:3/33: dread - d0/f4 zero size
2012-12-19T09:02:57.930 INFO:teuthology.task.workunit.client.0.out:0/20: chown d2/l4 0 1
2012-12-19T09:02:57.938 INFO:teuthology.task.workunit.client.0.out:6/13: dwrite f1 [0,4194304] 0
2012-12-19T09:02:57.941 INFO:teuthology.task.workunit.client.0.out:3/34: unlink d0/l2 0
2012-12-19T09:02:57.941 INFO:teuthology.task.workunit.client.0.out:2/38: creat d2/f5 x:0 0 0
2012-12-19T09:02:57.941 INFO:teuthology.task.workunit.client.0.out:5/15: write d1/f2 [919602,59236] 0
2012-12-19T09:02:57.945 INFO:teuthology.task.workunit.client.0.out:9/5: rename f0 to f1 0
2012-12-19T09:02:57.946 INFO:teuthology.task.workunit.client.0.out:9/6: readlink - no filename
2012-12-19T09:02:57.951 INFO:teuthology.task.workunit.client.0.out:1/15: truncate d0/f4 779328 0
2012-12-19T09:02:57.958 INFO:teuthology.task.workunit.client.0.out:0/21: rename l0 to d2/l8 0
2012-12-19T09:02:57.961 INFO:teuthology.task.workunit.client.0.out:2/39: mknod d2/c6 1
2012-12-19T09:02:57.961 INFO:teuthology.task.workunit.client.0.out:6/14: fsync f2 0
2012-12-19T09:02:57.963 INFO:teuthology.task.workunit.client.0.out:3/35: dread - d0/f4 zero size
2012-12-19T09:02:57.966 INFO:teuthology.task.workunit.client.0.out:5/16: write d1/f2 [1090452,100557] 0
2012-12-19T09:02:57.976 INFO:teuthology.task.workunit.client.0.out:0/22: creat d2/f9 x:0 0 0
2012-12-19T09:02:58.190 INFO:teuthology.task.workunit.client.0.out:1/16: rename d0/f2 to d0/f7 0
2012-12-19T09:02:58.192 INFO:teuthology.task.workunit.client.0.out:3/36: dread - d0/f4 zero size
2012-12-19T09:02:58.196 INFO:teuthology.task.workunit.client.0.out:7/3: dwrite f0 [0,4194304] 0
2012-12-19T09:02:58.199 INFO:teuthology.task.workunit.client.0.out:0/23: read - d2/f9 zero size
2012-12-19T09:02:58.213 INFO:teuthology.task.workunit.client.0.out:3/37: creat d0/f6 x:0 0 0
2012-12-19T09:02:58.281 INFO:teuthology.task.workunit.client.0.out:1/17: rename d0/f7 to d0/f8 0
2012-12-19T09:02:58.282 INFO:teuthology.task.workunit.client.0.out:7/4: creat f1 x:0 0 0
2012-12-19T09:02:58.288 INFO:teuthology.task.workunit.client.0.out:0/24: fdatasync d2/f9 0
2012-12-19T09:02:58.288 INFO:teuthology.task.workunit.client.0.out:4/14: dwrite f2 [0,4194304] 0
2012-12-19T09:02:58.295 INFO:teuthology.task.workunit.client.0.out:3/38: unlink d0/l1 0
2012-12-19T09:02:58.299 INFO:teuthology.task.workunit.client.0.out:1/18: mkdir d0/d9 0
2012-12-19T09:02:58.300 INFO:teuthology.task.workunit.client.0.out:7/5: creat f2 x:0 0 0
2012-12-19T09:02:58.301 INFO:teuthology.task.workunit.client.0.out:8/9: dwrite f2 [0,4194304] 0
2012-12-19T09:02:58.302 INFO:teuthology.task.workunit.client.0.out:4/15: mknod c4 1
2012-12-19T09:02:58.303 INFO:teuthology.task.workunit.client.0.out:7/6: mknod c3 1
2012-12-19T09:02:58.305 INFO:teuthology.task.workunit.client.0.out:7/7: rmdir - no directory
2012-12-19T09:02:58.310 INFO:teuthology.task.workunit.client.0.out:1/19: unlink d0/f3 0
2012-12-19T09:02:58.318 INFO:teuthology.task.workunit.client.0.out:0/25: write d2/f9 [76977,87339] 0
2012-12-19T09:02:58.563 INFO:teuthology.task.workunit.client.0.out:8/10: fdatasync f2 0
2012-12-19T09:02:58.564 INFO:teuthology.task.workunit.client.0.out:9/7: dread f1 [0,4194304] 0
2012-12-19T09:02:58.564 INFO:teuthology.task.workunit.client.0.out:4/16: creat f5 x:0 0 0
2012-12-19T09:02:58.570 INFO:teuthology.task.workunit.client.0.out:1/20: symlink d0/la 0
2012-12-19T09:02:58.573 INFO:teuthology.task.workunit.client.0.out:6/15: dread f1 [0,4194304] 0
2012-12-19T09:02:58.576 INFO:teuthology.task.workunit.client.0.out:4/17: rename f2 to f6 0
2012-12-19T09:02:58.577 INFO:teuthology.task.workunit.client.0.out:2/40: dwrite d2/f5 [0,4194304] 0
2012-12-19T09:02:58.594 INFO:teuthology.task.workunit.client.0.out:0/26: dread d2/f9 [0,4194304] 0
2012-12-19T09:02:58.613 INFO:teuthology.task.workunit.client.0.out:8/11: sync
2012-12-19T09:02:58.843 INFO:teuthology.task.workunit.client.0.out:1/21: link d0/f6 d0/fb 0
2012-12-19T09:02:58.849 INFO:teuthology.task.workunit.client.0.out:6/16: write f2 [243846,40732] 0
2012-12-19T09:02:58.865 INFO:teuthology.task.workunit.client.0.out:5/17: dwrite d1/f2 [0,4194304] 0
2012-12-19T09:02:58.869 INFO:teuthology.task.workunit.client.0.out:2/41: read d2/f5 [1575459,75841] 0
2012-12-19T09:02:58.872 INFO:teuthology.task.workunit.client.0.out:8/12: fdatasync f2 0
2012-12-19T09:02:58.876 INFO:teuthology.task.workunit.client.0.out:6/17: symlink l3 0
2012-12-19T09:02:58.878 INFO:teuthology.task.workunit.client.0.out:0/27: dread d2/f9 [0,4194304] 0
2012-12-19T09:02:58.921 INFO:teuthology.task.workunit.client.0.out:5/18: dread - d1/f4 zero size
2012-12-19T09:02:58.925 INFO:teuthology.task.workunit.client.0.out:8/13: read f2 [3590759,56881] 0
2012-12-19T09:02:58.927 INFO:teuthology.task.workunit.client.0.out:1/22: dread d0/f1 [0,4194304] 0
2012-12-19T09:02:58.928 INFO:teuthology.task.workunit.client.0.out:6/18: write f1 [3665,67125] 0
2012-12-19T09:02:59.158 INFO:teuthology.task.workunit.client.0.out:2/42: link d2/l4 d2/l7 0
2012-12-19T09:02:59.166 INFO:teuthology.task.workunit.client.0.out:3/39: dwrite d0/f4 [0,4194304] 0
2012-12-19T09:02:59.168 INFO:teuthology.task.workunit.client.0.out:0/28: link d2/l7 d2/la 0
2012-12-19T09:02:59.168 INFO:teuthology.task.workunit.client.0.out:6/19: creat f4 x:0 0 0
2012-12-19T09:02:59.168 INFO:teuthology.task.workunit.client.0.out:1/23: rename d0 to d0/d9/dc 22
2012-12-19T09:02:59.171 INFO:teuthology.task.workunit.client.0.out:8/14: rename f2 to f3 0
2012-12-19T09:02:59.176 INFO:teuthology.task.workunit.client.0.out:7/8: dwrite f1 [0,4194304] 0
2012-12-19T09:02:59.183 INFO:teuthology.task.workunit.client.0.out:0/29: mkdir d2/db 0
2012-12-19T09:02:59.518 INFO:teuthology.task.workunit.client.0.out:6/20: link f1 f5 0
2012-12-19T09:02:59.518 INFO:teuthology.task.workunit.client.0.out:8/15: creat f4 x:0 0 0
2012-12-19T09:02:59.521 INFO:teuthology.task.workunit.client.0.out:2/43: truncate d2/f5 2252754 0
2012-12-19T09:02:59.525 INFO:teuthology.task.workunit.client.0.out:9/8: dwrite f1 [0,4194304] 0
2012-12-19T09:02:59.538 INFO:teuthology.task.workunit.client.0.out:1/24: mkdir d0/d9/dd 0
2012-12-19T09:02:59.538 INFO:teuthology.task.workunit.client.0.out:9/9: readlink - no filename
2012-12-19T09:02:59.538 INFO:teuthology.task.workunit.client.0.out:3/40: creat d0/f7 x:0 0 0
2012-12-19T09:02:59.544 INFO:teuthology.task.workunit.client.0.out:7/9: creat f4 x:0 0 0
2012-12-19T09:02:59.545 INFO:teuthology.task.workunit.client.0.out:2/44: getdents d2 0
2012-12-19T09:02:59.545 INFO:teuthology.task.workunit.client.0.out:4/18: dread f6 [0,4194304] 0
2012-12-19T09:02:59.545 INFO:teuthology.task.workunit.client.0.out:8/16: write f4 [243174,36825] 0
2012-12-19T09:02:59.546 INFO:teuthology.task.workunit.client.0.out:9/10: getdents . 0
2012-12-19T09:02:59.547 INFO:teuthology.task.workunit.client.0.out:1/25: symlink d0/le 0
2012-12-19T09:02:59.548 INFO:teuthology.task.workunit.client.0.out:0/30: write d2/f9 [313906,3902] 0
2012-12-19T09:02:59.550 INFO:teuthology.task.workunit.client.0.out:3/41: symlink d0/l8 0
2012-12-19T09:02:59.557 INFO:teuthology.task.workunit.client.0.out:4/19: mkdir d7 0
2012-12-19T09:02:59.557 INFO:teuthology.task.workunit.client.0.out:7/10: write f2 [240497,63336] 0
2012-12-19T09:02:59.562 INFO:teuthology.task.workunit.client.0.out:0/31: readlink l1 0
2012-12-19T09:02:59.566 INFO:teuthology.task.workunit.client.0.out:3/42: chown d0/f4 9309 1
2012-12-19T09:02:59.567 INFO:teuthology.task.workunit.client.0.out:2/45: creat d2/f8 x:0 0 0
2012-12-19T09:02:59.570 INFO:teuthology.task.workunit.client.0.out:2/46: sync
2012-12-19T09:02:59.590 INFO:teuthology.task.workunit.client.0.out:1/26: read d0/f8 [452021,73535] 0
2012-12-19T09:02:59.595 INFO:teuthology.task.workunit.client.0.out:8/17: dread f4 [0,4194304] 0
2012-12-19T09:02:59.595 INFO:teuthology.task.workunit.client.0.out:7/11: fsync f4 0
2012-12-19T09:02:59.596 INFO:teuthology.task.workunit.client.0.out:7/12: readlink - no filename
2012-12-19T09:02:59.599 INFO:teuthology.task.workunit.client.0.out:4/20: truncate f5 566404 0
2012-12-19T09:02:59.603 INFO:teuthology.task.workunit.client.0.out:2/47: symlink d2/l9 0
2012-12-19T09:02:59.604 INFO:teuthology.task.workunit.client.0.out:3/43: dread - d0/f7 zero size
2012-12-19T09:02:59.604 INFO:teuthology.task.workunit.client.0.out:8/18: mknod c5 1
2012-12-19T09:02:59.604 INFO:teuthology.task.workunit.client.0.out:0/32: write d2/f9 [1111509,1427] 0
2012-12-19T09:02:59.609 INFO:teuthology.task.workunit.client.0.out:7/13: creat f5 x:0 0 0
2012-12-19T09:02:59.613 INFO:teuthology.task.workunit.client.0.out:1/27: rename d0 to d0/d9/dd/df 22
2012-12-19T09:02:59.613 INFO:teuthology.task.workunit.client.0.out:4/21: write f6 [3767117,20701] 0
2012-12-19T09:02:59.614 INFO:teuthology.task.workunit.client.0.out:2/48: mknod d2/ca 1
2012-12-19T09:02:59.614 INFO:teuthology.task.workunit.client.0.out:8/19: creat f6 x:0 0 0
2012-12-19T09:02:59.614 INFO:teuthology.task.workunit.client.0.out:3/44: mknod d0/c9 1
2012-12-19T09:02:59.616 INFO:teuthology.task.workunit.client.0.out:5/19: dwrite d1/f4 [0,4194304] 0
2012-12-19T09:02:59.617 INFO:teuthology.task.workunit.client.0.out:8/20: sync
2012-12-19T09:02:59.618 INFO:teuthology.task.workunit.client.0.out:4/22: stat d7 0
2012-12-19T09:02:59.619 INFO:teuthology.task.workunit.client.0.out:7/14: creat f6 x:0 0 0
2012-12-19T09:02:59.620 INFO:teuthology.task.workunit.client.0.out:3/45: stat d0/l8 0
2012-12-19T09:02:59.620 INFO:teuthology.task.workunit.client.0.out:0/33: write d2/f9 [1119369,75550] 0
2012-12-19T09:02:59.621 INFO:teuthology.task.workunit.client.0.out:8/21: dread - f6 zero size
2012-12-19T09:02:59.625 INFO:teuthology.task.workunit.client.0.out:3/46: chown d0 0 1
2012-12-19T09:02:59.627 INFO:teuthology.task.workunit.client.0.out:7/15: mkdir d7 0
2012-12-19T09:02:59.628 INFO:teuthology.task.workunit.client.0.out:4/23: fsync f5 0
2012-12-19T09:02:59.937 INFO:teuthology.task.workunit.client.0.out:5/20: fdatasync d1/f4 0
2012-12-19T09:02:59.939 INFO:teuthology.task.workunit.client.0.out:1/28: truncate d0/fb 48752 0
2012-12-19T09:02:59.942 INFO:teuthology.task.workunit.client.0.out:2/49: rename d2/f5 to d2/fb 0
2012-12-19T09:02:59.944 INFO:teuthology.task.workunit.client.0.out:3/47: mknod d0/ca 1
2012-12-19T09:02:59.944 INFO:teuthology.task.workunit.client.0.out:8/22: fsync f6 0
2012-12-19T09:02:59.947 INFO:teuthology.task.workunit.client.0.out:7/16: write f2 [101058,119307] 0
2012-12-19T09:02:59.947 INFO:teuthology.task.workunit.client.0.out:6/21: dwrite f1 [4194304,4194304] 0
2012-12-19T09:02:59.947 INFO:teuthology.task.workunit.client.0.out:5/21: chown d1/f2 15 1
2012-12-19T09:02:59.947 INFO:teuthology.task.workunit.client.0.out:4/24: write f6 [2808570,10247] 0
2012-12-19T09:03:00.056 INFO:teuthology.task.workunit.client.0.out:0/34: truncate d2/f9 1296763 0
2012-12-19T09:03:00.057 INFO:teuthology.task.workunit.client.0.out:3/48: mknod d0/d5/cb 1
2012-12-19T09:03:00.058 INFO:teuthology.task.workunit.client.0.out:2/50: write d2/f8 [734665,65649] 0
2012-12-19T09:03:00.058 INFO:teuthology.task.workunit.client.0.out:1/29: getdents d0/d9/dd 0
2012-12-19T09:03:00.058 INFO:teuthology.task.workunit.client.0.out:9/11: dread f1 [0,4194304] 0
2012-12-19T09:03:00.058 INFO:teuthology.task.workunit.client.0.out:7/17: mknod d7/c8 1
2012-12-19T09:03:00.059 INFO:teuthology.task.workunit.client.0.out:6/22: getdents . 0
2012-12-19T09:03:00.060 INFO:teuthology.task.workunit.client.0.out:5/22: symlink d1/l5 0
2012-12-19T09:03:00.060 INFO:teuthology.task.workunit.client.0.out:4/25: mknod d7/c8 1
2012-12-19T09:03:00.064 INFO:teuthology.task.workunit.client.0.out:0/35: fdatasync d2/f9 0
2012-12-19T09:03:00.064 INFO:teuthology.task.workunit.client.0.out:2/51: rename d2 to d2/dc 22
2012-12-19T09:03:00.073 INFO:teuthology.task.workunit.client.0.out:5/23: rename d1 to d1/d6 22
2012-12-19T09:03:00.073 INFO:teuthology.task.workunit.client.0.out:6/23: creat f6 x:0 0 0
2012-12-19T09:03:00.245 INFO:teuthology.task.workunit.client.0.out:7/18: truncate f4 293656 0
2012-12-19T09:03:00.249 INFO:teuthology.task.workunit.client.0.out:4/26: mkdir d7/d9 0
2012-12-19T09:03:00.249 INFO:teuthology.task.workunit.client.0.out:3/49: creat d0/d5/fc x:0 0 0
2012-12-19T09:03:00.249 INFO:teuthology.task.workunit.client.0.out:1/30: mknod d0/d9/dd/c10 1
2012-12-19T09:03:00.289 INFO:teuthology.task.workunit.client.0.out:2/52: fsync d2/f8 0
2012-12-19T09:03:00.297 INFO:teuthology.task.workunit.client.0.out:7/19: mkdir d7/d9 0
2012-12-19T09:03:00.298 INFO:teuthology.task.workunit.client.0.out:6/24: write f2 [153910,65232] 0
2012-12-19T09:03:00.306 INFO:teuthology.task.workunit.client.0.out:0/36: creat d2/db/fc x:0 0 0
2012-12-19T09:03:00.312 INFO:teuthology.task.workunit.client.0.out:4/27: creat d7/fa x:0 0 0
2012-12-19T09:03:00.326 INFO:teuthology.task.workunit.client.0.out:6/25: dread - f6 zero size
2012-12-19T09:03:00.336 INFO:teuthology.task.workunit.client.0.out:2/53: write d2/f8 [1468850,53797] 0
2012-12-19T09:03:00.340 INFO:teuthology.task.workunit.client.0.out:7/20: dread f2 [0,4194304] 0
2012-12-19T09:03:00.343 INFO:teuthology.task.workunit.client.0.out:0/37: mkdir d2/db/dd 0
2012-12-19T09:03:00.345 INFO:teuthology.task.workunit.client.0.out:3/50: truncate d0/d5/fc 912391 0
2012-12-19T09:03:00.374 INFO:teuthology.task.workunit.client.0.out:4/28: creat d7/fb x:0 0 0
2012-12-19T09:03:00.377 INFO:teuthology.task.workunit.client.0.out:6/26: unlink f5 0
2012-12-19T09:03:00.380 INFO:teuthology.task.workunit.client.0.out:2/54: stat d2/l3 0
2012-12-19T09:03:00.381 INFO:teuthology.task.workunit.client.0.out:6/27: chown l3 229388 1
2012-12-19T09:03:00.382 INFO:teuthology.task.workunit.client.0.out:7/21: getdents d7 0
2012-12-19T09:03:00.388 INFO:teuthology.task.workunit.client.0.out:4/29: creat d7/fc x:0 0 0
2012-12-19T09:03:00.568 INFO:teuthology.task.workunit.client.0.out:2/55: unlink d2/l4 0
2012-12-19T09:03:00.569 INFO:teuthology.task.workunit.client.0.out:0/38: write d2/db/fc [223413,33064] 0
2012-12-19T09:03:00.570 INFO:teuthology.task.workunit.client.0.out:7/22: dread - f6 zero size
2012-12-19T09:03:00.570 INFO:teuthology.task.workunit.client.0.out:3/51: write d0/d5/fc [1434473,93267] 0
2012-12-19T09:03:00.572 INFO:teuthology.task.workunit.client.0.out:6/28: read f2 [273513,118432] 0
2012-12-19T09:03:00.574 INFO:teuthology.task.workunit.client.0.out:8/23: dwrite f4 [0,4194304] 0
2012-12-19T09:03:00.577 INFO:teuthology.task.workunit.client.0.out:4/30: unlink d7/fa 0
2012-12-19T09:03:00.578 INFO:teuthology.task.workunit.client.0.out:6/29: readlink l3 0
2012-12-19T09:03:00.584 INFO:teuthology.task.workunit.client.0.out:8/24: readlink l1 0
2012-12-19T09:03:00.585 INFO:teuthology.task.workunit.client.0.out:9/12: dwrite f1 [0,4194304] 0
2012-12-19T09:03:00.585 INFO:teuthology.task.workunit.client.0.out:2/56: creat d2/fd x:0 0 0
2012-12-19T09:03:00.777 INFO:teuthology.task.workunit.client.0.out:7/23: truncate f0 5097656 0
2012-12-19T09:03:00.789 INFO:teuthology.task.workunit.client.0.out:8/25: readlink l1 0
2012-12-19T09:03:00.791 INFO:teuthology.task.workunit.client.0.out:9/13: stat f1 0
2012-12-19T09:03:00.830 INFO:teuthology.task.workunit.client.0.out:5/24: dread d1/f4 [0,4194304] 0
2012-12-19T09:03:00.832 INFO:teuthology.task.workunit.client.0.out:2/57: unlink d2/fd 0
2012-12-19T09:03:00.845 INFO:teuthology.task.workunit.client.0.out:8/26: dread - f6 zero size
2012-12-19T09:03:00.851 INFO:teuthology.task.workunit.client.0.out:1/31: dwrite d0/f1 [0,4194304] 0
2012-12-19T09:03:00.872 INFO:teuthology.task.workunit.client.0.out:9/14: creat f2 x:0 0 0
2012-12-19T09:03:00.879 INFO:teuthology.task.workunit.client.0.out:5/25: chown d1/f4 245439613 1
2012-12-19T09:03:00.881 INFO:teuthology.task.workunit.client.0.out:2/58: creat d2/fe x:0 0 0
2012-12-19T09:03:00.885 INFO:teuthology.task.workunit.client.0.out:8/27: read f3 [3834855,104360] 0
2012-12-19T09:03:00.899 INFO:teuthology.task.workunit.client.0.out:1/32: creat d0/f11 x:0 0 0
2012-12-19T09:03:00.899 INFO:teuthology.task.workunit.client.0.out:3/52: dread d0/d5/fc [0,4194304] 0
2012-12-19T09:03:00.902 INFO:teuthology.task.workunit.client.0.out:5/26: mkdir d1/d7 0
2012-12-19T09:03:00.910 INFO:teuthology.task.workunit.client.0.out:9/15: sync
2012-12-19T09:03:01.123 INFO:teuthology.task.workunit.client.0.out:2/59: truncate d2/fe 168042 0
2012-12-19T09:03:01.129 INFO:teuthology.task.workunit.client.0.out:9/16: mkdir d3 0
2012-12-19T09:03:01.314 INFO:teuthology.task.workunit.client.0.out:3/53: truncate d0/f7 800974 0
2012-12-19T09:03:01.320 INFO:teuthology.task.workunit.client.0.out:9/17: unlink f2 0
2012-12-19T09:03:01.333 INFO:teuthology.task.workunit.client.0.out:6/30: dread f1 [0,4194304] 0
2012-12-19T09:03:01.333 INFO:teuthology.task.workunit.client.0.out:9/18: read f1 [757459,24529] 0
2012-12-19T09:03:01.338 INFO:teuthology.task.workunit.client.0.out:4/31: dwrite d7/fc [0,4194304] 0
2012-12-19T09:03:01.339 INFO:teuthology.task.workunit.client.0.out:0/39: dwrite d2/db/fc [0,4194304] 0
2012-12-19T09:03:01.343 INFO:teuthology.task.workunit.client.0.out:7/24: dwrite f5 [0,4194304] 0
2012-12-19T09:03:01.354 INFO:teuthology.task.workunit.client.0.out:9/19: write f1 [4852284,83029] 0
2012-12-19T09:03:01.356 INFO:teuthology.task.workunit.client.0.out:7/25: dread - f6 zero size
2012-12-19T09:03:01.378 INFO:teuthology.task.workunit.client.0.out:9/20: rmdir d3 0
2012-12-19T09:03:01.424 INFO:teuthology.task.workunit.client.0.out:0/40: rename d2/db/fc to d2/db/fe 0
2012-12-19T09:03:01.428 INFO:teuthology.task.workunit.client.0.out:8/28: dwrite f4 [4194304,4194304] 0
2012-12-19T09:03:01.430 INFO:teuthology.task.workunit.client.0.out:7/26: dread f4 [0,4194304] 0
2012-12-19T09:03:01.432 INFO:teuthology.task.workunit.client.0.out:9/21: write f1 [1669867,46401] 0
2012-12-19T09:03:01.437 INFO:teuthology.task.workunit.client.0.out:1/33: dwrite d0/fb [0,4194304] 0
2012-12-19T09:03:01.440 INFO:teuthology.task.workunit.client.0.out:5/27: dwrite d1/f4 [4194304,4194304] 0
2012-12-19T09:03:01.445 INFO:teuthology.task.workunit.client.0.out:7/27: unlink f4 0
2012-12-19T09:03:01.450 INFO:teuthology.task.workunit.client.0.out:1/34: chown d0/f1 8747834 1
2012-12-19T09:03:01.463 INFO:teuthology.task.workunit.client.0.out:5/28: mknod d1/c8 1
2012-12-19T09:03:01.465 INFO:teuthology.task.workunit.client.0.out:0/41: creat d2/db/dd/ff x:0 0 0
2012-12-19T09:03:01.465 INFO:teuthology.task.workunit.client.0.out:2/60: dwrite d2/f8 [0,4194304] 0
2012-12-19T09:03:01.748 INFO:teuthology.task.workunit.client.0.out:7/28: truncate f5 2150536 0
2012-12-19T09:03:01.754 INFO:teuthology.task.workunit.client.0.out:5/29: chown d1/l5 1 1
2012-12-19T09:03:01.754 INFO:teuthology.task.workunit.client.0.out:1/35: fdatasync d0/f8 0
2012-12-19T09:03:01.754 INFO:teuthology.task.workunit.client.0.out:3/54: dwrite d0/f6 [0,4194304] 0
2012-12-19T09:03:01.756 INFO:teuthology.task.workunit.client.0.out:2/61: getdents d2 0
2012-12-19T09:03:01.757 INFO:teuthology.task.workunit.client.0.out:0/42: getdents d2/db 0
2012-12-19T09:03:01.757 INFO:teuthology.task.workunit.client.0.out:7/29: mknod d7/d9/ca 1
2012-12-19T09:03:01.758 INFO:teuthology.task.workunit.client.0.out:5/30: readlink d1/l5 0
2012-12-19T09:03:01.760 INFO:teuthology.task.workunit.client.0.out:4/32: dread f6 [0,4194304] 0
2012-12-19T09:03:01.762 INFO:teuthology.task.workunit.client.0.out:6/31: dwrite f1 [8388608,4194304] 0
2012-12-19T09:03:01.768 INFO:teuthology.task.workunit.client.0.out:1/36: write d0/fb [1181813,24357] 0
2012-12-19T09:03:01.772 INFO:teuthology.task.workunit.client.0.out:7/30: truncate f0 239340 0
2012-12-19T09:03:01.772 INFO:teuthology.task.workunit.client.0.out:3/55: write d0/f6 [1150833,79755] 0
2012-12-19T09:03:02.070 INFO:teuthology.task.workunit.client.0.out:7/31: stat f1 0
2012-12-19T09:03:02.076 INFO:teuthology.task.workunit.client.0.out:4/33: dread - d7/fb zero size
2012-12-19T09:03:02.077 INFO:teuthology.task.workunit.client.0.out:0/43: dread - d2/db/dd/ff zero size
2012-12-19T09:03:02.077 INFO:teuthology.task.workunit.client.0.out:6/32: read f1 [3177014,93475] 0
2012-12-19T09:03:02.090 INFO:teuthology.task.workunit.client.0.out:3/56: chown d0/l3 326738534 1
2012-12-19T09:03:02.093 INFO:teuthology.task.workunit.client.0.out:7/32: mknod d7/d9/cb 1
2012-12-19T09:03:02.094 INFO:teuthology.task.workunit.client.0.out:6/33: dread - f4 zero size
2012-12-19T09:03:02.130 INFO:teuthology.task.workunit.client.0.out:6/34: chown f2 52641 1
2012-12-19T09:03:02.131 INFO:teuthology.task.workunit.client.0.out:9/22: dread f1 [0,4194304] 0
2012-12-19T09:03:02.131 INFO:teuthology.task.workunit.client.0.out:8/29: dwrite f6 [0,4194304] 0
2012-12-19T09:03:02.140 INFO:teuthology.task.workunit.client.0.out:9/23: fsync f1 0
2012-12-19T09:03:02.141 INFO:teuthology.task.workunit.client.0.out:3/57: dread d0/f7 [0,4194304] 0
2012-12-19T09:03:02.147 INFO:teuthology.task.workunit.client.0.out:8/30: truncate f3 1317285 0
2012-12-19T09:03:02.151 INFO:teuthology.task.workunit.client.0.out:8/31: chown f3 16153 1
2012-12-19T09:03:02.158 INFO:teuthology.task.workunit.client.0.out:9/24: truncate f1 183306 0
2012-12-19T09:03:02.187 INFO:teuthology.task.workunit.client.0.out:2/62: dwrite d2/fe [0,4194304] 0
2012-12-19T09:03:02.404 INFO:teuthology.task.workunit.client.0.out:8/32: truncate f6 4935988 0
2012-12-19T09:03:02.405 INFO:teuthology.task.workunit.client.0.out:5/31: dread d1/f2 [0,4194304] 0
2012-12-19T09:03:02.410 INFO:teuthology.task.workunit.client.0.out:3/58: rename d0/f6 to d0/d5/fd 0
2012-12-19T09:03:02.411 INFO:teuthology.task.workunit.client.0.out:9/25: dread f1 [0,4194304] 0
2012-12-19T09:03:02.415 INFO:teuthology.task.workunit.client.0.out:8/33: symlink l7 0
2012-12-19T09:03:02.419 INFO:teuthology.task.workunit.client.0.out:5/32: rmdir d1 39
2012-12-19T09:03:02.432 INFO:teuthology.task.workunit.client.0.out:8/34: symlink l8 0
2012-12-19T09:03:02.435 INFO:teuthology.task.workunit.client.0.out:2/63: creat d2/ff x:0 0 0
2012-12-19T09:03:02.437 INFO:teuthology.task.workunit.client.0.out:5/33: stat d1/d7 0
2012-12-19T09:03:02.438 INFO:teuthology.task.workunit.client.0.out:1/37: dwrite d0/f6 [0,4194304] 0
2012-12-19T09:03:02.827 INFO:teuthology.task.workunit.client.0.out:3/59: write d0/f7 [464434,66600] 0
2012-12-19T09:03:02.827 INFO:teuthology.task.workunit.client.0.out:8/35: readlink l7 0
2012-12-19T09:03:02.828 INFO:teuthology.task.workunit.client.0.out:9/26: dread f1 [0,4194304] 0
2012-12-19T09:03:02.830 INFO:teuthology.task.workunit.client.0.out:8/36: chown l7 98 1
2012-12-19T09:03:02.832 INFO:teuthology.task.workunit.client.0.out:4/34: dwrite d7/fc [0,4194304] 0
2012-12-19T09:03:02.832 INFO:teuthology.task.workunit.client.0.out:0/44: dwrite d2/f9 [0,4194304] 0
2012-12-19T09:03:02.834 INFO:teuthology.task.workunit.client.0.out:2/64: symlink d2/l10 0
2012-12-19T09:03:02.835 INFO:teuthology.task.workunit.client.0.out:9/27: mknod c4 1
2012-12-19T09:03:02.836 INFO:teuthology.task.workunit.client.0.out:7/33: dwrite f6 [0,4194304] 0
2012-12-19T09:03:02.836 INFO:teuthology.task.workunit.client.0.out:8/37: chown l8 244 1
2012-12-19T09:03:02.848 INFO:teuthology.task.workunit.client.0.out:9/28: chown f1 3852991 1
2012-12-19T09:03:02.850 INFO:teuthology.task.workunit.client.0.out:3/60: creat d0/fe x:0 0 0
2012-12-19T09:03:02.850 INFO:teuthology.task.workunit.client.0.out:8/38: chown f6 7 1
2012-12-19T09:03:02.850 INFO:teuthology.task.workunit.client.0.out:5/34: creat d1/d7/f9 x:0 0 0
2012-12-19T09:03:02.854 INFO:teuthology.task.workunit.client.0.out:2/65: dread - d2/ff zero size
2012-12-19T09:03:02.854 INFO:teuthology.task.workunit.client.0.out:6/35: dwrite f2 [0,4194304] 0
2012-12-19T09:03:02.859 INFO:teuthology.task.workunit.client.0.out:9/29: mkdir d5 0
2012-12-19T09:03:02.863 INFO:teuthology.task.workunit.client.0.out:1/38: truncate d0/f11 97537 0
2012-12-19T09:03:02.863 INFO:teuthology.task.workunit.client.0.out:7/34: mknod d7/cc 1
2012-12-19T09:03:02.871 INFO:teuthology.task.workunit.client.0.out:3/61: mknod d0/cf 1
2012-12-19T09:03:02.871 INFO:teuthology.task.workunit.client.0.out:5/35: chown d1/f4 330191804 1
2012-12-19T09:03:02.871 INFO:teuthology.task.workunit.client.0.out:0/45: creat d2/db/f10 x:0 0 0
2012-12-19T09:03:02.875 INFO:teuthology.task.workunit.client.0.out:9/30: fsync f1 0
2012-12-19T09:03:02.877 INFO:teuthology.task.workunit.client.0.out:6/36: mkdir d7 0
2012-12-19T09:03:02.877 INFO:teuthology.task.workunit.client.0.out:1/39: readlink d0/le 0
2012-12-19T09:03:03.161 INFO:teuthology.task.workunit.client.0.out:2/66: rename l1 to d2/l11 0
2012-12-19T09:03:03.182 INFO:teuthology.task.workunit.client.0.out:5/36: creat d1/d7/fa x:0 0 0
2012-12-19T09:03:03.182 INFO:teuthology.task.workunit.client.0.out:3/62: write d0/d5/fc [861860,31572] 0
2012-12-19T09:03:03.184 INFO:teuthology.task.workunit.client.0.out:1/40: mkdir d0/d9/d12 0
2012-12-19T09:03:03.184 INFO:teuthology.task.workunit.client.0.out:0/46: creat d2/db/f11 x:0 0 0
2012-12-19T09:03:03.204 INFO:teuthology.task.workunit.client.0.out:2/67: creat d2/f12 x:0 0 0
2012-12-19T09:03:03.206 INFO:teuthology.task.workunit.client.0.out:0/47: mknod d2/c12 1
2012-12-19T09:03:03.209 INFO:teuthology.task.workunit.client.0.out:5/37: fsync d1/d7/fa 0
2012-12-19T09:03:03.209 INFO:teuthology.task.workunit.client.0.out:1/41: write d0/f4 [217735,62974] 0
2012-12-19T09:03:03.210 INFO:teuthology.task.workunit.client.0.out:3/63: fsync d0/d5/fd 0
2012-12-19T09:03:03.211 INFO:teuthology.task.workunit.client.0.out:7/35: dread f5 [0,4194304] 0
2012-12-19T09:03:03.217 INFO:teuthology.task.workunit.client.0.out:3/64: chown d0/f4 104 1
2012-12-19T09:03:03.378 INFO:teuthology.task.workunit.client.0.out:0/48: link d2/l4 d2/db/l13 0
2012-12-19T09:03:03.379 INFO:teuthology.task.workunit.client.0.out:1/42: mknod d0/d9/d12/c13 1
2012-12-19T09:03:03.388 INFO:teuthology.task.workunit.client.0.out:7/36: creat d7/d9/fd x:0 0 0
2012-12-19T09:03:03.389 INFO:teuthology.task.workunit.client.0.out:3/65: getdents d0 0
2012-12-19T09:03:03.389 INFO:teuthology.task.workunit.client.0.out:4/35: dwrite d7/fb [0,4194304] 0
2012-12-19T09:03:03.700 INFO:teuthology.task.workunit.client.0.out:5/38: link d1/d7/fa d1/d7/fb 0
2012-12-19T09:03:03.702 INFO:teuthology.task.workunit.client.0.out:8/39: dwrite f4 [4194304,4194304] 0
2012-12-19T09:03:03.705 INFO:teuthology.task.workunit.client.0.out:0/49: dread - d2/db/dd/ff zero size
2012-12-19T09:03:03.715 INFO:teuthology.task.workunit.client.0.out:5/39: mkdir d1/dc 0
2012-12-19T09:03:03.721 INFO:teuthology.task.workunit.client.0.out:1/43: creat d0/d9/d12/f14 x:0 0 0
2012-12-19T09:03:03.722 INFO:teuthology.task.workunit.client.0.out:4/36: creat d7/fd x:0 0 0
2012-12-19T09:03:03.722 INFO:teuthology.task.workunit.client.0.out:7/37: creat d7/d9/fe x:0 0 0
2012-12-19T09:03:03.722 INFO:teuthology.task.workunit.client.0.out:6/37: dwrite f1 [0,4194304] 0
2012-12-19T09:03:03.723 INFO:teuthology.task.workunit.client.0.out:9/31: dwrite f1 [0,4194304] 0
2012-12-19T09:03:03.733 INFO:teuthology.task.workunit.client.0.out:9/32: readlink - no filename
2012-12-19T09:03:03.739 INFO:teuthology.task.workunit.client.0.out:5/40: readlink d1/l5 0
2012-12-19T09:03:03.758 INFO:teuthology.task.workunit.client.0.out:0/50: fsync d2/f9 0
2012-12-19T09:03:03.764 INFO:teuthology.task.workunit.client.0.out:4/37: truncate f6 5153472 0
2012-12-19T09:03:03.765 INFO:teuthology.task.workunit.client.0.out:7/38: chown d7/d9/fd 155171403 1
2012-12-19T09:03:03.766 INFO:teuthology.task.workunit.client.0.out:9/33: write f1 [138345,56300] 0
2012-12-19T09:03:03.801 INFO:teuthology.task.workunit.client.0.out:1/44: creat d0/d9/d12/f15 x:0 0 0
2012-12-19T09:03:03.802 INFO:teuthology.task.workunit.client.0.out:0/51: chown d2/db/fe 2496 1
2012-12-19T09:03:04.012 INFO:teuthology.task.workunit.client.0.out:5/41: truncate d1/f4 7232062 0
2012-12-19T09:03:04.150 INFO:teuthology.task.workunit.client.0.out:7/39: truncate f1 3614374 0
2012-12-19T09:03:04.152 INFO:teuthology.task.workunit.client.0.out:9/34: write f1 [891545,21848] 0
2012-12-19T09:03:04.153 INFO:teuthology.task.workunit.client.0.out:2/68: dwrite d2/f12 [0,4194304] 0
2012-12-19T09:03:04.157 INFO:teuthology.task.workunit.client.0.out:4/38: rename d7/fb to d7/fe 0
2012-12-19T09:03:04.164 INFO:teuthology.task.workunit.client.0.out:5/42: rmdir d1/d7 39
2012-12-19T09:03:04.170 INFO:teuthology.task.workunit.client.0.out:4/39: stat f5 0
2012-12-19T09:03:04.188 INFO:teuthology.task.workunit.client.0.out:7/40: dread - d7/d9/fe zero size
2012-12-19T09:03:04.194 INFO:teuthology.task.workunit.client.0.out:9/35: mkdir d5/d6 0
2012-12-19T09:03:04.194 INFO:teuthology.task.workunit.client.0.out:7/41: readlink - no filename
2012-12-19T09:03:04.199 INFO:teuthology.task.workunit.client.0.out:9/36: chown d5 54834582 1
2012-12-19T09:03:04.204 INFO:teuthology.task.workunit.client.0.out:0/52: truncate d2/db/f11 223939 0
2012-12-19T09:03:04.205 INFO:teuthology.task.workunit.client.0.out:4/40: rename d7 to d7/d9/df 22
2012-12-19T09:03:04.248 INFO:teuthology.task.workunit.client.0.out:5/43: dread - d1/d7/f9 zero size
2012-12-19T09:03:04.251 INFO:teuthology.task.workunit.client.0.out:3/66: dwrite d0/f4 [0,4194304] 0
2012-12-19T09:03:04.252 INFO:teuthology.task.workunit.client.0.out:8/40: dread f6 [0,4194304] 0
2012-12-19T09:03:12.673 INFO:teuthology.task.workunit.client.0.out:9/37: truncate f1 2844851 0
2012-12-19T09:03:12.678 INFO:teuthology.task.workunit.client.0.out:4/41: read - d7/fd zero size
2012-12-19T09:03:12.678 INFO:teuthology.task.workunit.client.0.out:7/42: write d7/d9/fe [993702,51104] 0
2012-12-19T09:03:12.689 INFO:teuthology.task.workunit.client.0.out:8/41: symlink l9 0
2012-12-19T09:03:12.690 INFO:teuthology.task.workunit.client.0.out:5/44: creat d1/fd x:0 0 0
2012-12-19T09:03:12.692 INFO:teuthology.task.workunit.client.0.out:6/38: dwrite f4 [0,4194304] 0
2012-12-19T09:03:12.759 INFO:teuthology.task.workunit.client.0.out:9/38: write f1 [2428963,58654] 0
2012-12-19T09:03:12.765 INFO:teuthology.task.workunit.client.0.out:3/67: creat d0/d5/f10 x:0 0 0
2012-12-19T09:03:12.766 INFO:teuthology.task.workunit.client.0.out:6/39: chown d7 268167 1
2012-12-19T09:03:12.774 INFO:teuthology.task.workunit.client.0.out:7/43: mkdir d7/d9/df 0
2012-12-19T09:03:12.776 INFO:teuthology.task.workunit.client.0.out:8/42: truncate f4 6434371 0
2012-12-19T09:03:12.779 INFO:teuthology.task.workunit.client.0.out:9/39: unlink f1 0
2012-12-19T09:03:12.780 INFO:teuthology.task.workunit.client.0.out:4/42: creat d7/d9/f10 x:0 0 0
2012-12-19T09:03:12.794 INFO:teuthology.task.workunit.client.0.out:6/40: mknod d7/c8 1
2012-12-19T09:03:12.797 INFO:teuthology.task.workunit.client.0.out:8/43: unlink l8 0
2012-12-19T09:03:12.798 INFO:teuthology.task.workunit.client.0.out:7/44: mknod d7/c10 1
2012-12-19T09:03:12.801 INFO:teuthology.task.workunit.client.0.out:1/45: dread d0/f6 [0,4194304] 0
2012-12-19T09:03:12.808 INFO:teuthology.task.workunit.client.0.out:9/40: getdents d5/d6 0
2012-12-19T09:03:12.809 INFO:teuthology.task.workunit.client.0.out:9/41: read - no filename
2012-12-19T09:03:12.809 INFO:teuthology.task.workunit.client.0.out:9/42: write - no filename
2012-12-19T09:03:12.809 INFO:teuthology.task.workunit.client.0.out:6/41: getdents d7 0
2012-12-19T09:03:12.818 INFO:teuthology.task.workunit.client.0.out:8/44: truncate f3 1236365 0
2012-12-19T09:03:12.951 INFO:teuthology.task.workunit.client.0.out:8/45: mknod ca 1
2012-12-19T09:03:12.951 INFO:teuthology.task.workunit.client.0.out:6/42: rename d7 to d7/d9 22
2012-12-19T09:03:12.953 INFO:teuthology.task.workunit.client.0.out:1/46: symlink d0/l16 0
2012-12-19T09:03:12.954 INFO:teuthology.task.workunit.client.0.out:7/45: creat d7/d9/f11 x:0 0 0
2012-12-19T09:03:12.958 INFO:teuthology.task.workunit.client.0.out:9/43: symlink d5/d6/l7 0
2012-12-19T09:03:12.959 INFO:teuthology.task.workunit.client.0.out:2/69: dwrite d2/f8 [0,4194304] 0
2012-12-19T09:03:13.098 INFO:teuthology.task.workunit.client.0.out:1/47: rmdir d0 39
2012-12-19T09:03:13.169 INFO:teuthology.task.workunit.client.0.out:6/43: creat d7/fa x:0 0 0
2012-12-19T09:03:13.170 INFO:teuthology.task.workunit.client.0.out:9/44: rmdir d5/d6 39
2012-12-19T09:03:13.170 INFO:teuthology.task.workunit.client.0.out:9/45: dread - no filename
2012-12-19T09:03:13.173 INFO:teuthology.task.workunit.client.0.out:7/46: dread f2 [0,4194304] 0
2012-12-19T09:03:13.259 INFO:teuthology.task.workunit.client.0.out:2/70: fsync d2/f8 0
2012-12-19T09:03:13.262 INFO:teuthology.task.workunit.client.0.out:0/53: dwrite d2/f9 [0,4194304] 0
2012-12-19T09:03:13.265 INFO:teuthology.task.workunit.client.0.out:1/48: write d0/f8 [1254955,18155] 0
2012-12-19T09:03:13.267 INFO:teuthology.task.workunit.client.0.out:9/46: mkdir d5/d8 0
2012-12-19T09:03:13.268 INFO:teuthology.task.workunit.client.0.out:9/47: fdatasync - no filename
2012-12-19T09:03:13.268 INFO:teuthology.task.workunit.client.0.out:6/44: write f2 [334546,60574] 0
2012-12-19T09:03:13.275 INFO:teuthology.task.workunit.client.0.out:7/47: stat d7/d9/fe 0
2012-12-19T09:03:13.276 INFO:teuthology.task.workunit.client.0.out:6/45: chown f6 152200348 1
2012-12-19T09:03:13.276 INFO:teuthology.task.workunit.client.0.out:9/48: mknod d5/c9 1
2012-12-19T09:03:13.276 INFO:teuthology.task.workunit.client.0.out:9/49: fsync - no filename
2012-12-19T09:03:13.277 INFO:teuthology.task.workunit.client.0.out:0/54: rmdir d2/db 39
2012-12-19T09:03:13.278 INFO:teuthology.task.workunit.client.0.out:1/49: stat d0/f6 0
2012-12-19T09:03:13.280 INFO:teuthology.task.workunit.client.0.out:5/45: dread d1/f2 [0,4194304] 0
2012-12-19T09:03:13.496 INFO:teuthology.task.workunit.client.0.out:7/48: truncate f1 1888575 0
2012-12-19T09:03:13.497 INFO:teuthology.task.workunit.client.0.out:3/68: dread d0/d5/fd [0,4194304] 0
2012-12-19T09:03:13.499 INFO:teuthology.task.workunit.client.0.out:6/46: truncate f6 103181 0
2012-12-19T09:03:13.499 INFO:teuthology.task.workunit.client.0.out:9/50: rename d5 to d5/d8/da 22
2012-12-19T09:03:13.499 INFO:teuthology.task.workunit.client.0.out:9/51: dwrite - no filename
2012-12-19T09:03:13.500 INFO:teuthology.task.workunit.client.0.out:9/52: write - no filename
2012-12-19T09:03:13.500 INFO:teuthology.task.workunit.client.0.out:1/50: fdatasync d0/f11 0
2012-12-19T09:03:13.501 INFO:teuthology.task.workunit.client.0.out:4/43: dwrite d7/fe [0,4194304] 0
2012-12-19T09:03:13.501 INFO:teuthology.task.workunit.client.0.out:7/49: chown d7/d9 316483 1
2012-12-19T09:03:13.504 INFO:teuthology.task.workunit.client.0.out:6/47: mkdir d7/db 0
2012-12-19T09:03:13.507 INFO:teuthology.task.workunit.client.0.out:3/69: mknod d0/d5/c11 1
2012-12-19T09:03:13.507 INFO:teuthology.task.workunit.client.0.out:6/48: stat l3 0
2012-12-19T09:03:13.508 INFO:teuthology.task.workunit.client.0.out:1/51: read d0/f8 [324174,80099] 0
2012-12-19T09:03:13.511 INFO:teuthology.task.workunit.client.0.out:0/55: rename d2/db/f10 to d2/db/dd/f14 0
2012-12-19T09:03:13.512 INFO:teuthology.task.workunit.client.0.out:9/53: rename d5/d8 to d5/d8/db 22
2012-12-19T09:03:13.513 INFO:teuthology.task.workunit.client.0.out:6/49: chown l3 76253511 1
2012-12-19T09:03:13.513 INFO:teuthology.task.workunit.client.0.out:4/44: getdents d7/d9 0
2012-12-19T09:03:13.514 INFO:teuthology.task.workunit.client.0.out:8/46: dwrite f4 [0,4194304] 0
2012-12-19T09:03:13.520 INFO:teuthology.task.workunit.client.0.out:4/45: chown d7/d9 458 1
2012-12-19T09:03:13.523 INFO:teuthology.task.workunit.client.0.out:1/52: unlink d0/f4 0
2012-12-19T09:03:13.523 INFO:teuthology.task.workunit.client.0.out:3/70: creat d0/d5/f12 x:0 0 0
2012-12-19T09:03:13.524 INFO:teuthology.task.workunit.client.0.out:8/47: stat l1 0
2012-12-19T09:03:13.524 INFO:teuthology.task.workunit.client.0.out:4/46: chown d7 53113404 1
2012-12-19T09:03:13.544 INFO:teuthology.task.workunit.client.0.out:4/47: read d7/fc [2275674,24972] 0
2012-12-19T09:03:13.552 INFO:teuthology.task.workunit.client.0.out:2/71: dwrite d2/f12 [0,4194304] 0
2012-12-19T09:03:13.608 INFO:teuthology.task.workunit.client.0.out:8/48: sync
2012-12-19T09:03:13.608 INFO:teuthology.task.workunit.client.0.out:9/54: sync
2012-12-19T09:03:13.608 INFO:teuthology.task.workunit.client.0.out:9/55: fsync - no filename
2012-12-19T09:03:13.608 INFO:teuthology.task.workunit.client.0.out:5/46: sync
2012-12-19T09:03:13.910 INFO:teuthology.task.workunit.client.0.out:0/56: link d2/db/dd/ff d2/db/dd/f15 0
2012-12-19T09:03:13.910 INFO:teuthology.task.workunit.client.0.out:1/53: creat d0/d9/d12/f17 x:0 0 0
2012-12-19T09:03:13.916 INFO:teuthology.task.workunit.client.0.out:4/48: fdatasync f5 0
2012-12-19T09:03:13.922 INFO:teuthology.task.workunit.client.0.out:1/54: symlink d0/l18 0
2012-12-19T09:03:13.927 INFO:teuthology.task.workunit.client.0.out:8/49: rename f3 to fb 0
2012-12-19T09:03:13.928 INFO:teuthology.task.workunit.client.0.out:2/72: fdatasync d2/fb 0
2012-12-19T09:03:13.931 INFO:teuthology.task.workunit.client.0.out:9/56: symlink d5/d6/lc 0
2012-12-19T09:03:14.124 INFO:teuthology.task.workunit.client.0.out:5/47: rename d1/d7 to d1/de 0
2012-12-19T09:03:14.126 INFO:teuthology.task.workunit.client.0.out:1/55: symlink d0/l19 0
2012-12-19T09:03:14.128 INFO:teuthology.task.workunit.client.0.out:8/50: write f4 [2335379,54958] 0
2012-12-19T09:03:14.135 INFO:teuthology.task.workunit.client.0.out:4/49: rename f6 to d7/f11 0
2012-12-19T09:03:14.141 INFO:teuthology.task.workunit.client.0.out:7/50: dwrite f2 [0,4194304] 0
2012-12-19T09:03:14.141 INFO:teuthology.task.workunit.client.0.out:9/57: creat d5/fd x:0 0 0
2012-12-19T09:03:14.141 INFO:teuthology.task.workunit.client.0.out:5/48: stat d1/fd 0
2012-12-19T09:03:14.142 INFO:teuthology.task.workunit.client.0.out:2/73: creat d2/f13 x:0 0 0
2012-12-19T09:03:14.152 INFO:teuthology.task.workunit.client.0.out:0/57: rename d2/l8 to d2/db/dd/l16 0
2012-12-19T09:03:14.162 INFO:teuthology.task.workunit.client.0.out:1/56: sync
2012-12-19T09:03:14.349 INFO:teuthology.task.workunit.client.0.out:8/51: write fb [527924,17680] 0
2012-12-19T09:03:14.350 INFO:teuthology.task.workunit.client.0.out:4/50: creat d7/f12 x:0 0 0
2012-12-19T09:03:14.353 INFO:teuthology.task.workunit.client.0.out:2/74: dread - d2/ff zero size
2012-12-19T09:03:14.356 INFO:teuthology.task.workunit.client.0.out:5/49: mkdir d1/dc/df 0
2012-12-19T09:03:14.356 INFO:teuthology.task.workunit.client.0.out:7/51: write f2 [3514258,104842] 0
2012-12-19T09:03:14.357 INFO:teuthology.task.workunit.client.0.out:9/58: write d5/fd [423138,107301] 0
2012-12-19T09:03:14.358 INFO:teuthology.task.workunit.client.0.out:1/57: readlink d0/l19 0
2012-12-19T09:03:14.359 INFO:teuthology.task.workunit.client.0.out:6/50: dwrite f4 [0,4194304] 0
2012-12-19T09:03:14.360 INFO:teuthology.task.workunit.client.0.out:8/52: read f4 [3895893,90879] 0
2012-12-19T09:03:14.363 INFO:teuthology.task.workunit.client.0.out:4/51: read d7/fe [3892090,23096] 0
2012-12-19T09:03:14.365 INFO:teuthology.task.workunit.client.0.out:3/71: dwrite d0/f7 [0,4194304] 0
2012-12-19T09:03:14.369 INFO:teuthology.task.workunit.client.0.out:9/59: mkdir d5/de 0
2012-12-19T09:03:14.369 INFO:teuthology.task.workunit.client.0.out:5/50: dread - d1/de/f9 zero size
2012-12-19T09:03:14.373 INFO:teuthology.task.workunit.client.0.out:4/52: chown d7 2 1
2012-12-19T09:03:14.377 INFO:teuthology.task.workunit.client.0.out:0/58: truncate d2/db/fe 2335842 0
2012-12-19T09:03:14.378 INFO:teuthology.task.workunit.client.0.out:8/53: fsync f4 0
2012-12-19T09:03:14.378 INFO:teuthology.task.workunit.client.0.out:1/58: stat d0/d9/d12/f15 0
2012-12-19T09:03:14.601 INFO:teuthology.task.workunit.client.0.out:7/52: write d7/d9/fd [546211,117421] 0
2012-12-19T09:03:14.601 INFO:teuthology.task.workunit.client.0.out:6/51: write f1 [4574132,2438] 0
2012-12-19T09:03:14.602 INFO:teuthology.task.workunit.client.0.out:3/72: rmdir d0 39
2012-12-19T09:03:14.606 INFO:teuthology.task.workunit.client.0.out:7/53: stat f1 0
2012-12-19T09:03:14.608 INFO:teuthology.task.workunit.client.0.out:4/53: dread - d7/f12 zero size
2012-12-19T09:03:14.608 INFO:teuthology.task.workunit.client.0.out:5/51: mknod d1/dc/c10 1
2012-12-19T09:03:14.610 INFO:teuthology.task.workunit.client.0.out:1/59: mkdir d0/d1a 0
2012-12-19T09:03:14.610 INFO:teuthology.task.workunit.client.0.out:6/52: fdatasync f2 0
2012-12-19T09:03:14.612 INFO:teuthology.task.workunit.client.0.out:0/59: symlink d2/db/l17 0
2012-12-19T09:03:14.614 INFO:teuthology.task.workunit.client.0.out:3/73: dread - d0/fe zero size
2012-12-19T09:03:14.618 INFO:teuthology.task.workunit.client.0.out:1/60: mknod d0/d9/c1b 1
2012-12-19T09:03:14.623 INFO:teuthology.task.workunit.client.0.out:6/53: write f2 [4845046,91705] 0
2012-12-19T09:03:14.625 INFO:teuthology.task.workunit.client.0.out:0/60: mkdir d2/db/d18 0
2012-12-19T09:03:14.626 INFO:teuthology.task.workunit.client.0.out:7/54: creat d7/d9/df/f12 x:0 0 0
2012-12-19T09:03:14.628 INFO:teuthology.task.workunit.client.0.out:4/54: truncate d7/f11 5502884 0
2012-12-19T09:03:14.634 INFO:teuthology.task.workunit.client.0.out:3/74: creat d0/d5/f13 x:0 0 0
2012-12-19T09:03:14.638 INFO:teuthology.task.workunit.client.0.out:5/52: rename d1/dc/df to d1/dc/d11 0
2012-12-19T09:03:14.640 INFO:teuthology.task.workunit.client.0.out:4/55: getdents d7 0
2012-12-19T09:03:14.642 INFO:teuthology.task.workunit.client.0.out:0/61: chown d2/db/dd/f14 50736029 1
2012-12-19T09:03:14.642 INFO:teuthology.task.workunit.client.0.out:3/75: readlink d0/l8 0
2012-12-19T09:03:14.651 INFO:teuthology.task.workunit.client.0.out:1/61: rename d0/l19 to d0/d9/d12/l1c 0
2012-12-19T09:03:14.654 INFO:teuthology.task.workunit.client.0.out:0/62: readlink d2/la 0
2012-12-19T09:03:14.656 INFO:teuthology.task.workunit.client.0.out:2/75: dwrite d2/f8 [0,4194304] 0
2012-12-19T09:03:14.659 INFO:teuthology.task.workunit.client.0.out:3/76: symlink d0/d5/l14 0
2012-12-19T09:03:14.874 INFO:teuthology.task.workunit.client.0.out:8/54: dread f4 [0,4194304] 0
2012-12-19T09:03:14.874 INFO:teuthology.task.workunit.client.0.out:9/60: dwrite d5/fd [0,4194304] 0
2012-12-19T09:03:14.909 INFO:teuthology.task.workunit.client.0.out:8/55: mkdir dc 0
2012-12-19T09:03:14.919 INFO:teuthology.task.workunit.client.0.out:9/61: symlink d5/de/lf 0
2012-12-19T09:03:14.957 INFO:teuthology.task.workunit.client.0.out:8/56: symlink dc/ld 0
2012-12-19T09:03:14.962 INFO:teuthology.task.workunit.client.0.out:6/54: dread f1 [8388608,4194304] 0
2012-12-19T09:03:14.967 INFO:teuthology.task.workunit.client.0.out:8/57: chown dc/ld 0 1
2012-12-19T09:03:14.967 INFO:teuthology.task.workunit.client.0.out:3/77: dread d0/d5/fc [0,4194304] 0
2012-12-19T09:03:14.968 INFO:teuthology.task.workunit.client.0.out:9/62: creat d5/f10 x:0 0 0
2012-12-19T09:03:14.972 INFO:teuthology.task.workunit.client.0.out:9/63: sync
2012-12-19T09:03:14.976 INFO:teuthology.task.workunit.client.0.out:8/58: fdatasync f6 0
2012-12-19T09:03:14.977 INFO:teuthology.task.workunit.client.0.out:5/53: dread d1/f4 [0,4194304] 0
2012-12-19T09:03:14.978 INFO:teuthology.task.workunit.client.0.out:7/55: dwrite d7/d9/fd [0,4194304] 0
2012-12-19T09:03:14.980 INFO:teuthology.task.workunit.client.0.out:7/56: readlink - no filename
2012-12-19T09:03:14.983 INFO:teuthology.task.workunit.client.0.out:9/64: dread - d5/f10 zero size
2012-12-19T09:03:14.984 INFO:teuthology.task.workunit.client.0.out:8/59: mknod dc/ce 1
2012-12-19T09:03:14.985 INFO:teuthology.task.workunit.client.0.out:4/56: dwrite d7/fd [0,4194304] 0
2012-12-19T09:03:14.986 INFO:teuthology.task.workunit.client.0.out:1/62: dread d0/fb [0,4194304] 0
2012-12-19T09:03:15.036 INFO:teuthology.task.workunit.client.0.out:7/57: write f2 [1540325,81643] 0
2012-12-19T09:03:15.038 INFO:teuthology.task.workunit.client.0.out:1/63: sync
2012-12-19T09:03:15.041 INFO:teuthology.task.workunit.client.0.out:3/78: truncate d0/d5/fc 1204659 0
2012-12-19T09:03:15.041 INFO:teuthology.task.workunit.client.0.out:8/60: rename dc to dc/df 22
2012-12-19T09:03:15.041 INFO:teuthology.task.workunit.client.0.out:2/76: dread d2/f12 [0,4194304] 0
2012-12-19T09:03:15.042 INFO:teuthology.task.workunit.client.0.out:1/64: sync
2012-12-19T09:03:15.043 INFO:teuthology.task.workunit.client.0.out:0/63: dwrite d2/f9 [4194304,4194304] 0
2012-12-19T09:03:15.573 INFO:teuthology.task.workunit.client.0.out:4/57: creat d7/f13 x:0 0 0
2012-12-19T09:03:15.573 INFO:teuthology.task.workunit.client.0.out:5/54: link d1/f2 d1/dc/d11/f12 0
2012-12-19T09:03:15.574 INFO:teuthology.task.workunit.client.0.out:7/58: getdents d7 0
2012-12-19T09:03:15.798 INFO:teuthology.task.workunit.client.0.out:9/65: link d5/fd d5/d6/f11 0
2012-12-19T09:03:16.016 INFO:teuthology.task.workunit.client.0.out:8/61: link l9 dc/l10 0
2012-12-19T09:03:16.016 INFO:teuthology.task.workunit.client.0.out:3/79: write d0/f4 [3016509,47821] 0
2012-12-19T09:03:16.022 INFO:teuthology.task.workunit.client.0.out:4/58: dread - d7/f12 zero size
2012-12-19T09:03:16.022 INFO:teuthology.task.workunit.client.0.out:0/64: fdatasync d2/db/f11 0
2012-12-19T09:03:16.025 INFO:teuthology.task.workunit.client.0.out:9/66: unlink d5/f10 0
2012-12-19T09:03:16.027 INFO:teuthology.task.workunit.client.0.out:8/62: symlink dc/l11 0
2012-12-19T09:03:16.030 INFO:teuthology.task.workunit.client.0.out:7/59: fsync d7/d9/df/f12 0
2012-12-19T09:03:16.032 INFO:teuthology.task.workunit.client.0.out:3/80: write d0/f4 [4878844,31868] 0
2012-12-19T09:03:16.035 INFO:teuthology.task.workunit.client.0.out:0/65: chown d2/db/f11 213832769 1
2012-12-19T09:03:16.175 INFO:teuthology.task.workunit.client.0.out:9/67: write d5/fd [1888943,6734] 0
2012-12-19T09:03:16.409 INFO:teuthology.task.workunit.client.0.out:4/59: link d7/f13 d7/f14 0
2012-12-19T09:03:16.412 INFO:teuthology.task.workunit.client.0.out:3/81: rmdir d0 39
2012-12-19T09:03:16.413 INFO:teuthology.task.workunit.client.0.out:6/55: dwrite f1 [4194304,4194304] 0
2012-12-19T09:03:16.419 INFO:teuthology.task.workunit.client.0.out:0/66: unlink d2/db/f11 0
2012-12-19T09:03:16.421 INFO:teuthology.task.workunit.client.0.out:4/60: mknod d7/d9/c15 1
2012-12-19T09:03:16.426 INFO:teuthology.task.workunit.client.0.out:7/60: rename d7/d9/fd to d7/f13 0
2012-12-19T09:03:16.426 INFO:teuthology.task.workunit.client.0.out:9/68: getdents d5/d6 0
2012-12-19T09:03:16.427 INFO:teuthology.task.workunit.client.0.out:3/82: dread - d0/d5/f12 zero size
2012-12-19T09:03:16.428 INFO:teuthology.task.workunit.client.0.out:6/56: read - d7/fa zero size
2012-12-19T09:03:16.431 INFO:teuthology.task.workunit.client.0.out:4/61: chown d7/fc 1624714 1
2012-12-19T09:03:16.439 INFO:teuthology.task.workunit.client.0.out:0/67: unlink d2/db/dd/l16 0
2012-12-19T09:03:16.441 INFO:teuthology.task.workunit.client.0.out:6/57: rmdir d7 39
2012-12-19T09:03:16.467 INFO:teuthology.task.workunit.client.0.out:9/69: fsync d5/fd 0
2012-12-19T09:03:16.663 INFO:teuthology.task.workunit.client.0.out:4/62: dread - d7/f12 zero size
2012-12-19T09:03:16.670 INFO:teuthology.task.workunit.client.0.out:0/68: rmdir d2/db 39
2012-12-19T09:03:16.671 INFO:teuthology.task.workunit.client.0.out:2/77: dwrite d2/fe [0,4194304] 0
2012-12-19T09:03:16.672 INFO:teuthology.task.workunit.client.0.out:6/58: creat d7/fc x:0 0 0
2012-12-19T09:03:16.679 INFO:teuthology.task.workunit.client.0.out:5/55: dwrite d1/f2 [0,4194304] 0
2012-12-19T09:03:16.681 INFO:teuthology.task.workunit.client.0.out:9/70: creat d5/d6/f12 x:0 0 0
2012-12-19T09:03:16.682 INFO:teuthology.task.workunit.client.0.out:1/65: dwrite d0/d9/d12/f17 [0,4194304] 0
2012-12-19T09:03:16.683 INFO:teuthology.task.workunit.client.0.out:4/63: rmdir d7/d9 39
2012-12-19T09:03:16.694 INFO:teuthology.task.workunit.client.0.out:3/83: truncate d0/d5/fd 829960 0
2012-12-19T09:03:16.698 INFO:teuthology.task.workunit.client.0.out:2/78: mkdir d2/d14 0
2012-12-19T09:03:16.699 INFO:teuthology.task.workunit.client.0.out:0/69: chown d2/db/dd/f15 783132 1
2012-12-19T09:03:16.699 INFO:teuthology.task.workunit.client.0.out:8/63: dread f4 [0,4194304] 0
2012-12-19T09:03:16.699 INFO:teuthology.task.workunit.client.0.out:5/56: readlink d1/l5 0
2012-12-19T09:03:16.702 INFO:teuthology.task.workunit.client.0.out:1/66: unlink d0/l18 0
2012-12-19T09:03:16.708 INFO:teuthology.task.workunit.client.0.out:9/71: mkdir d5/de/d13 0
2012-12-19T09:03:16.708 INFO:teuthology.task.workunit.client.0.out:0/70: chown d2/l7 243563258 1
2012-12-19T09:03:16.710 INFO:teuthology.task.workunit.client.0.out:2/79: symlink d2/l15 0
2012-12-19T09:03:16.711 INFO:teuthology.task.workunit.client.0.out:8/64: fdatasync fb 0
2012-12-19T09:03:16.713 INFO:teuthology.task.workunit.client.0.out:1/67: rmdir d0/d9 39
2012-12-19T09:03:16.717 INFO:teuthology.task.workunit.client.0.out:4/64: rename d7/f14 to d7/d9/f16 0
2012-12-19T09:03:16

Actions #1

Updated by Sam Lang over 11 years ago

  • Description updated (diff)
  • Status changed from New to Can't reproduce

I ran this test throughout the day yesterday and couldn't reproduce it, with message delays enabled. Marking as can't reproduce for now. If it pops up again, we should look more closely at the ceph fuse process, the queue length of the objecter read wait queue, etc.

Actions #2

Updated by Greg Farnum almost 8 years ago

  • Component(FS) Client added
Actions

Also available in: Atom PDF