Actions
Bug #3547
closedrbd: suites/iozone.sh failure
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I get a strange hang when running the workunit "suites/iozone.sh" when it is run after an "rbd:" task in teuthology. It looks like this by default creates an ext3 file system on an 10GB rbd image. I have the result of one of these failures below. I am sure that others I've seen are similar (possibly identical) to this but I don't have evidence of it right now. The iozone workunit is simple: iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1 iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1 iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 Here is the meaning of the flags: -c include close() in timing calculations -e include fsync() and fflush() in timing -s 1024M created file is size 1024 megabytes -r 16K record size is 16 kilobytes -t 1 use one thread while testing -F f1 created file is named "f1" -i 0 perform write tests -i 1 (then) perform read tests In the case below, the first two tests passed. The last one hung after displaying these messages (full detail later): INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3 ... INFO:teuthology.task.workunit.client.0.out: Min xfer = 0.00 KB INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:Child 0 INFO:teuthology.task.workunit.client.0.err:f3: No such file or directory Perhaps the error writing block 9407 led to the failure to create the output file "f3". I don't know why progress on the test stopped at that point, however. It may be that it fails because we're attempting to create a 10GB file in an ext3 file system that sits on a 10GB backing disk. I think a teuthology yaml file that ends with this should reproduce the problem: tasks: - ceph: branch: master - rbd: all: - workunit: clients: all: - suites/iozone.sh Here is a more complete listing of the messages from the last iozone run in "suites/iozone.sh": INFO:teuthology.task.workunit.client.0.out: Iozone: Performance Test of File I/O INFO:teuthology.task.workunit.client.0.out: Version $Revision: 3.397 $ INFO:teuthology.task.workunit.client.0.out: Compiled for 64 bit mode. INFO:teuthology.task.workunit.client.0.out: Build: linux-AMD64 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins INFO:teuthology.task.workunit.client.0.out: Al Slater, Scott Rhine, Mike Wisner, Ken Goss INFO:teuthology.task.workunit.client.0.out: Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, INFO:teuthology.task.workunit.client.0.out: Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, INFO:teuthology.task.workunit.client.0.out: Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, INFO:teuthology.task.workunit.client.0.out: Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, INFO:teuthology.task.workunit.client.0.out: Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. INFO:teuthology.task.workunit.client.0.out: Ben England. INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: Run began: Wed Nov 28 12:16:35 2012 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: Include close in write timing INFO:teuthology.task.workunit.client.0.out: Include fsync in write timing INFO:teuthology.task.workunit.client.0.out: File size set to 10485760 KB INFO:teuthology.task.workunit.client.0.out: Record Size 1024 KB INFO:teuthology.task.workunit.client.0.out: Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1 INFO:teuthology.task.workunit.client.0.out: Output is in Kbytes/sec INFO:teuthology.task.workunit.client.0.out: Time Resolution = 0.000001 seconds. INFO:teuthology.task.workunit.client.0.out: Processor cache size set to 1024 Kbytes. INFO:teuthology.task.workunit.client.0.out: Processor cache line size set to 32 bytes. INFO:teuthology.task.workunit.client.0.out: File stride size set to 17 * record size. INFO:teuthology.task.workunit.client.0.out: Throughput test with 1 process INFO:teuthology.task.workunit.client.0.out: Each process writes a 10485760 Kbyte file in 1024 Kbyte records INFO:teuthology.task.ceph.mon.c.err:2012-11-28 12:18:28.679554 7f33b6e17700 -1 mon.c@2(peon).paxos(logm updating c 4066..4568) lease_expire from mon.0 10.214.132.20:6789/0 is 0.017564 seconds in the past; mons are laggy or clocks are too skewed INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3 INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out: Children see throughput for 1 initial writers = 0.00 KB/sec INFO:teuthology.task.workunit.client.0.out: Parent sees throughput for 1 initial writers = 0.00 KB/sec INFO:teuthology.task.workunit.client.0.out: Min throughput per process = 0.00 KB/sec INFO:teuthology.task.workunit.client.0.out: Max throughput per process = 0.00 KB/sec INFO:teuthology.task.workunit.client.0.out: Avg throughput per process = 0.00 KB/sec INFO:teuthology.task.workunit.client.0.out: Min xfer = 0.00 KB INFO:teuthology.task.workunit.client.0.out: INFO:teuthology.task.workunit.client.0.out:Child 0 INFO:teuthology.task.workunit.client.0.err:f3: No such file or directory
Actions