Ceph » rbd

I get a strange hang when running the workunit "suites/iozone.sh" 
when it is run after an "rbd:" task in teuthology.  It looks like
this by default creates an ext3 file system on an 10GB rbd image.

I have the result of one of these failures below.  I am sure that
others I've seen are similar (possibly identical) to this but I
don't have evidence of it right now.

The iozone workunit is simple:

    iozone -c -e -s 1024M -r 16K -t 1 -F f1 -i 0 -i 1
    iozone -c -e -s 1024M -r 1M -t 1 -F f2 -i 0 -i 1
    iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1

Here is the meaning of the flags:

    -c        include close() in timing calculations
    -e        include fsync() and fflush() in timing
    -s 1024M  created file is size 1024 megabytes
    -r 16K    record size is 16 kilobytes
    -t 1      use one thread while testing
    -F f1     created file is named "f1" 
    -i 0      perform write tests
    -i 1      (then) perform read tests

In the case below, the first two tests passed.  The last one
hung after displaying these messages (full detail later):

    INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3
    ...
    INFO:teuthology.task.workunit.client.0.out:    Min xfer             =       0.00 KB
    INFO:teuthology.task.workunit.client.0.out:
    INFO:teuthology.task.workunit.client.0.out:Child 0
    INFO:teuthology.task.workunit.client.0.err:f3: No such file or directory

Perhaps the error writing block 9407 led to the failure to
create the output file "f3".  I don't know why progress on the
test stopped at that point, however.

It may be that it fails because we're attempting to create a 10GB
file in an ext3 file system that sits on a 10GB backing disk.

I think a teuthology yaml file that ends with this should reproduce
the problem:

    tasks:
    - ceph:
        branch: master
    - rbd:
        all:
    - workunit:
        clients:
          all:
             - suites/iozone.sh

Here is a more complete listing of the messages from the last
iozone run in "suites/iozone.sh":

INFO:teuthology.task.workunit.client.0.out:    Iozone: Performance Test of File I/O
INFO:teuthology.task.workunit.client.0.out:            Version $Revision: 3.397 $
INFO:teuthology.task.workunit.client.0.out:        Compiled for 64 bit mode.
INFO:teuthology.task.workunit.client.0.out:        Build: linux-AMD64
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:    Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
INFO:teuthology.task.workunit.client.0.out:                 Al Slater, Scott Rhine, Mike Wisner, Ken Goss
INFO:teuthology.task.workunit.client.0.out:                 Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
INFO:teuthology.task.workunit.client.0.out:                 Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
INFO:teuthology.task.workunit.client.0.out:                 Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
INFO:teuthology.task.workunit.client.0.out:                 Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
INFO:teuthology.task.workunit.client.0.out:                 Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
INFO:teuthology.task.workunit.client.0.out:                 Ben England.
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:    Run began: Wed Nov 28 12:16:35 2012
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:    Include close in write timing
INFO:teuthology.task.workunit.client.0.out:    Include fsync in write timing
INFO:teuthology.task.workunit.client.0.out:    File size set to 10485760 KB
INFO:teuthology.task.workunit.client.0.out:    Record Size 1024 KB
INFO:teuthology.task.workunit.client.0.out:    Command line used: iozone -c -e -s 10240M -r 1M -t 1 -F f3 -i 0 -i 1
INFO:teuthology.task.workunit.client.0.out:    Output is in Kbytes/sec
INFO:teuthology.task.workunit.client.0.out:    Time Resolution = 0.000001 seconds.
INFO:teuthology.task.workunit.client.0.out:    Processor cache size set to 1024 Kbytes.
INFO:teuthology.task.workunit.client.0.out:    Processor cache line size set to 32 bytes.
INFO:teuthology.task.workunit.client.0.out:    File stride size set to 17 * record size.
INFO:teuthology.task.workunit.client.0.out:    Throughput test with 1 process
INFO:teuthology.task.workunit.client.0.out:    Each process writes a 10485760 Kbyte file in 1024 Kbyte records
INFO:teuthology.task.ceph.mon.c.err:2012-11-28 12:18:28.679554 7f33b6e17700 -1 mon.c@2(peon).paxos(logm updating c 4066..4568) lease_expire from mon.0 10.214.132.20:6789/0 is 0.017564 seconds in the past; mons are laggy or clocks are too skewed
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:Error writing block 9407, fd= 3
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:    Children see throughput for  1 initial writers     =       0.00 KB/sec
INFO:teuthology.task.workunit.client.0.out:    Parent sees throughput for  1 initial writers     =       0.00 KB/sec
INFO:teuthology.task.workunit.client.0.out:    Min throughput per process     =       0.00 KB/sec
INFO:teuthology.task.workunit.client.0.out:    Max throughput per process     =       0.00 KB/sec
INFO:teuthology.task.workunit.client.0.out:    Avg throughput per process     =       0.00 KB/sec
INFO:teuthology.task.workunit.client.0.out:    Min xfer             =       0.00 KB
INFO:teuthology.task.workunit.client.0.out:
INFO:teuthology.task.workunit.client.0.out:Child 0
INFO:teuthology.task.workunit.client.0.err:f3: No such file or directory