Tools » teuthology

The result is a job takes over 2 hours to terminate and get marked "dead."

2015-03-04T19:17:00.172 INFO:tasks.thrashosds.thrasher:Moving pg 3.4s1 from osd.2 to osd.3
2015-03-04T19:17:00.172 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@burnupi04.front.sepia.ceph.com to ubuntu@burnupi17.front.sepia.ceph.com
2015-03-04T19:17:04.679 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:17:01.928951 7fe61c257700  0 -- 10.214.133.12:0/1008759 >> 10.214.133.12:6810/7659 pipe(0x7fe60c02a390 sd=12 :36542 s=2 pgs=74 cs=1 l=1 c=0x7fe60c019cc0).injecting socket failure
2015-03-04T19:18:29.749 ERROR:paramiko.transport:Exception: Remote transport is ignoring rekey requests
2015-03-04T19:18:29.768 ERROR:paramiko.transport:Traceback (most recent call last):
2015-03-04T19:18:29.768 ERROR:paramiko.transport:  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1528, in run
2015-03-04T19:18:29.768 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2015-03-04T19:18:29.768 ERROR:paramiko.transport:  File "/usr/lib/python2.7/dist-packages/paramiko/packet.py", line 378, in read_message
2015-03-04T19:18:29.768 ERROR:paramiko.transport:    raise SSHException('Remote transport is ignoring rekey requests')
2015-03-04T19:18:29.769 ERROR:paramiko.transport:SSHException: Remote transport is ignoring rekey requests
2015-03-04T19:18:29.769 ERROR:paramiko.transport:
2015-03-04T19:18:29.785 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.788649 7fe617fff700  0 monclient: hunting for new mon
2015-03-04T19:18:29.798 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.801231 7fe603fe2700  0 -- 10.214.133.12:0/1008759 >> 10.214.134.18:6805/56832 pipe(0x7fe60c01b150 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe60c019e60).fault
2015-03-04T19:18:29.798 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.801454 7fe61c156700  0 -- 10.214.133.12:0/1008759 >> 10.214.134.18:6810/57820 pipe(0x7fe60c022870 sd=12 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe60c00b300).fault
2015-03-04T21:08:23.570 ERROR:teuthology.run_tasks:Manager failed: radosbench
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/radosbench.py", line 80, in task
    run.wait(radosbench.itervalues(), timeout=timeout)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 395, in wait
    check_time()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 134, in __call__
    raise MaxWhileTries(error_msg)
MaxWhileTries: reached maximum tries (1500) after waiting for 9000 seconds
2015-03-04T21:08:23.701 INFO:tasks.thrashosds:joining thrashosds
2015-03-04T21:08:23.702 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/thrashosds.py", line 183, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/ceph_manager.py", line 353, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
EOFError