Ceph

Run: http://pulpito.ceph.com/teuthology-2017-02-26_19:25:18-upgrade:kraken-x-master-distro-basic-vps/
Jobs: many
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-02-26_19:25:18-upgrade:kraken-x-master-distro-basic-vps/862339/teuthology.log

2017-02-26T20:33:23.027 INFO:tasks.ceph.ceph_manager.ceph:pg 3.9 in trimming, state: active+clean+snaptrim
2017-02-26T20:33:23.027 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting
2017-02-26T20:33:33.029 CRITICAL:root:  File "gevent.libev.corecext.pyx", line 367, in gevent.libev.corecext.loop.handle_error (src/gevent/libev/gevent.corecext.c:5051)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 558, in handle_error
    self.print_exception(context, type, value, tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 605, in print_exception
    traceback.print_exception(type, value, tb, file=errstream)
  File "/usr/lib/python2.7/traceback.py", line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File "/usr/lib/python2.7/traceback.py", line 13, in _print
    file.write(str+terminator)

2017-02-26T20:33:33.029 CRITICAL:root:IOError
2017-02-26T20:33:33.030 DEBUG:teuthology.parallel:result is None
2017-02-26T20:33:33.030 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 63, in _run_spawned
    mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/full_sequential.py", line 37, in task
    mgr.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 262, in task
    running.get()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 482, in get
    self._raise_exception()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 159, in _raise_exception
    reraise(*self.exc_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/rados.py", line 253, in thread
    manager.wait_snap_trimming_complete(pool);
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1666, in wait_snap_trimming_complete
    'failed to complete snap trimming before timeout'
AssertionError: failed to complete snap trimming before timeout