Ceph

If I run the "suites/tiobench.sh" suite with kclient I sometimes
(or maybe always) get a failure, after the suite has completed.

Here's a short yaml file that should reproduce it:

roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2]
- [mon.b, mds.a, osd.3, osd.4, osd.5]
- [client.0]
kernel:
  mon:
    branch: testing
  client:
    branch: testing
tasks:
- install:
- ceph:
- kclient:
- workunit:
    clients:
      all:
        - suites/tiobench.sh

Below is an output excerpt that demonstrates the problem.

INFO:teuthology.task.workunit.client.0.err:Thread: 18977 chose seek of 1982464
INFO:teuthology.task.workunit.client.0.err:Thread: 18977 chose seek of 16560128
INFO:teuthology.task.workunit.client.0.err:Thread: 18977 chose seek of 8736768
INFO:teuthology.task.workunit.client.0.err:Thread: 18977 chose seek of 19906560
INFO:teuthology.task.workunit.client.0.err:Done!
INFO:teuthology.task.workunit.client.0.err:+ echo OK.
INFO:teuthology.task.workunit.client.0.out:Created 10 threads
INFO:teuthology.task.workunit.client.0.out:Created 10 threads
INFO:teuthology.task.workunit.client.0.out:Created 10 threads
INFO:teuthology.task.workunit.client.0.out:Created 10 threads
INFO:teuthology.task.workunit.client.0.out:write:200.00000,5.11579,0.36000,2.84000,0.05890,335.89900,0.00000,0.00000
INFO:teuthology.task.workunit.client.0.out:rwrite:39.06250,8.72664,3.02000,2.69000,0.00739,0.04900,0.00000,0.00000
INFO:teuthology.task.workunit.client.0.out:read:200.00000,0.14649,6.79000,1.50000,0.00311,10.04400,0.00000,0.00000
INFO:teuthology.task.workunit.client.0.out:rread:39.06250,1.83596,3.99000,0.99000,0.00496,0.09700,0.00000,0.00000
INFO:teuthology.task.workunit.client.0.out:total:0.02695,335.89900,0.00000,0.00000
INFO:teuthology.task.workunit.client.0.out:OK.
INFO:teuthology.task.workunit:Stopping suites/tiobench.sh on client.0...
<type 'exceptions.StopIteration'>
INFO:teuthology.orchestra.run.err:rm: cannot remove `/tmp/cephtest/mnt.0/client.0': Permission denied
INFO:teuthology.orchestra.run.err:rmdir: failed to remove `/tmp/cephtest/mnt.0': Device or resource busy
INFO:teuthology.task.kclient:Unmounting kernel clients...
INFO:teuthology.misc:Shutting down mds daemons...
INFO:teuthology.task.ceph.mds.a:Stopped
INFO:teuthology.misc:Shutting down osd daemons...
INFO:teuthology.task.ceph.osd.1:Stopped
INFO:teuthology.task.ceph.osd.0:Stopped