Ceph

Verified by two runs:

http://pulpito.ceph.com/teuthology-2016-06-13_09:48:57-powercycle-jewel-testing-basic-mira/
job: 256858

http://pulpito.ceph.com/teuthology-2016-06-13_13:58:29-powercycle-jewel-testing-basic-smithi/
job: 256948

Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-13_13:58:29-powercycle-jewel-testing-basic-smithi/256948/teuthology.log

2016-06-13T16:32:43.894 INFO:teuthology.orchestra.run.smithi056:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2016-06-13T16:32:43.997 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0...
2016-06-13T16:32:43.997 INFO:teuthology.orchestra.run.smithi056:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/workunit.client.0 /home/ubuntu/cephtest/clone'
2016-06-13T16:32:44.318 DEBUG:teuthology.parallel:result is None
2016-06-13T16:32:44.318 INFO:teuthology.orchestra.run.smithi056:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0'
2016-06-13T16:32:44.347 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0
2016-06-13T16:32:44.347 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2016-06-13T16:32:46.073 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2016-06-13T16:32:46.073 INFO:teuthology.orchestra.run.smithi056:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0" 
2016-06-13T16:32:46.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2016-06-13T16:32:46.087 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi056.front.sepia.ceph.com...
2016-06-13T16:32:46.087 INFO:teuthology.orchestra.run.smithi056:Running: 'sudo fusermount -u /home/ubuntu/cephtest/mnt.0'
2016-06-13T16:32:46.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi056.stderr:ceph-fuse[14972]: fuse finished with error 0 and tester_r 0
2016-06-13T16:32:46.211 INFO:teuthology.orchestra.run.smithi056:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0" 
2016-06-13T16:32:46.222 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3'
2016-06-13T16:32:46.222 INFO:teuthology.orchestra.run:waiting for 900
2016-06-13T16:32:52.224 INFO:teuthology.orchestra.run.smithi056:Running: 'rmdir -- /home/ubuntu/cephtest/mnt.0'
2016-06-13T16:32:52.235 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds
2016-06-13T16:32:52.273 INFO:tasks.thrashosds:joining thrashosds
2016-06-13T16:32:52.273 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 139, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/thrashosds.py", line 184, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 431, in do_join
    self.thread.get()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 454, in get
    self._raise_exception()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 171, in _raise_exception
    reraise(*self.exc_info)
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 646, in wrapper
    return func(self)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 693, in do_thrash
    timeout=self.config.get('timeout')
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 1673, in wait_for_recovery
    'failed to recover before timeout expired'
AssertionError: failed to recover before timeout expired