Ceph

Run: http://pulpito.ceph.com/teuthology-2017-03-08_03:25:22-upgrade:kraken-x-master-distro-basic-smithi/
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-03-08_03:25:22-upgrade:kraken-x-master-distro-basic-smithi/894894/teuthology.log

2017-03-08T08:51:37.873 INFO:tasks.ceph.osd.0.smithi091.stderr:2017-03-08 08:51:37.875189 7f43d8c72700 -1 received  signal: Hangup from  PID: 527636 task name: /usr/bin/python /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  UID: 0
2017-03-08T08:51:37.877 INFO:teuthology.orchestra.run.smithi091.stderr:osd.5: osd_enable_op_tracker = 'false'
2017-03-08T08:51:37.896 INFO:teuthology.orchestra.run.smithi091:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops'
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:no valid command found; 10 closest matches:
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush rule rm <name>
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush tree
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush rule create-simple <name> <root> <type> {firstn|indep}
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush rule create-erasure <name> {<profile>}
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush get-tunable straw_calc_version
2017-03-08T08:51:37.913 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush show-tunables
2017-03-08T08:51:37.914 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush tunables legacy|argonaut|bobtail|firefly|hammer|jewel|optimal|default
2017-03-08T08:51:37.914 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush set-tunable straw_calc_version <int>
2017-03-08T08:51:37.914 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush reweight <name> <float[0.0-]>
2017-03-08T08:51:37.914 INFO:teuthology.orchestra.run.smithi091.stderr:osd crush reweight-subtree <name> <float[0.0-]>
2017-03-08T08:51:37.914 INFO:teuthology.orchestra.run.smithi091.stderr:Error EINVAL: invalid command
2017-03-08T08:51:37.922 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 754, in wrapper
    return func(self)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 871, in do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 503, in thrash_cluster_full
    self.ceph_manager.raw_cluster_cmd('osd', 'set-full-ratio', '.001')
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1008, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 403, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)
CommandFailedError: Command failed on smithi091 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set-full-ratio .001'

2017-03-08T08:51:37.924 CRITICAL:root:  File "gevent.libev.corecext.pyx", line 367, in gevent.libev.corecext.loop.handle_error (src/gevent/libev/gevent.corecext.c:5051)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 558, in handle_error
    self.print_exception(context, type, value, tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 605, in print_exception
    traceback.print_exception(type, value, tb, file=errstream)
  File "/usr/lib/python2.7/traceback.py", line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File "/usr/lib/python2.7/traceback.py", line 13, in _print
    file.write(str+terminator)

2017-03-08T08:51:37.924 CRITICAL:root:IOError