Bug #11035
Updated by Zack Cerza almost 9 years ago
The result is a job takes over 2 hours to terminate and get marked "dead." <pre> 2015-03-04T19:17:00.172 INFO:tasks.thrashosds.thrasher:Moving pg 3.4s1 from osd.2 to osd.3 2015-03-04T19:17:00.172 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@burnupi04.front.sepia.ceph.com to ubuntu@burnupi17.front.sepia.ceph.com 2015-03-04T19:17:04.679 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:17:01.928951 7fe61c257700 0 -- 10.214.133.12:0/1008759 >> 10.214.133.12:6810/7659 pipe(0x7fe60c02a390 sd=12 :36542 s=2 pgs=74 cs=1 l=1 c=0x7fe60c019cc0).injecting socket failure 2015-03-04T19:18:29.749 ERROR:paramiko.transport:Exception: Remote transport is ignoring rekey requests 2015-03-04T19:18:29.768 ERROR:paramiko.transport:Traceback (most recent call last): 2015-03-04T19:18:29.768 ERROR:paramiko.transport: File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1528, in run 2015-03-04T19:18:29.768 ERROR:paramiko.transport: ptype, m = self.packetizer.read_message() 2015-03-04T19:18:29.768 ERROR:paramiko.transport: File "/usr/lib/python2.7/dist-packages/paramiko/packet.py", line 378, in read_message 2015-03-04T19:18:29.768 ERROR:paramiko.transport: raise SSHException('Remote transport is ignoring rekey requests') 2015-03-04T19:18:29.769 ERROR:paramiko.transport:SSHException: Remote transport is ignoring rekey requests 2015-03-04T19:18:29.769 ERROR:paramiko.transport: 2015-03-04T19:18:29.785 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.788649 7fe617fff700 0 monclient: hunting for new mon 2015-03-04T19:18:29.798 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.801231 7fe603fe2700 0 -- 10.214.133.12:0/1008759 >> 10.214.134.18:6805/56832 pipe(0x7fe60c01b150 sd=10 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe60c019e60).fault 2015-03-04T19:18:29.798 INFO:tasks.radosbench.radosbench.0.burnupi04.stderr:2015-03-04 19:18:29.801454 7fe61c156700 0 -- 10.214.133.12:0/1008759 >> 10.214.134.18:6810/57820 pipe(0x7fe60c022870 sd=12 :0 s=1 pgs=0 cs=0 l=1 c=0x7fe60c00b300).fault 2015-03-04T21:08:23.570 ERROR:teuthology.run_tasks:Manager failed: radosbench Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/radosbench.py", line 80, in task run.wait(radosbench.itervalues(), timeout=timeout) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 395, in wait check_time() File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) MaxWhileTries: reached maximum tries (1500) after waiting for 9000 seconds 2015-03-04T21:08:23.701 INFO:tasks.thrashosds:joining thrashosds 2015-03-04T21:08:23.702 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/thrashosds.py", line 183, in task thrash_proc.do_join() File "/var/lib/teuthworker/src/ceph-qa-suite_wip-cot-firefly/tasks/ceph_manager.py", line 353, in do_join self.thread.get() File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get raise self._exception EOFError </pre>