Tools » teuthology

2020-04-06T08:14:40.860 INFO:teuthology.orchestra.console:Performing hard reset of smithi205
2020-04-06T08:14:40.893 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi205.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power reset
2020-04-06T08:14:40.917 INFO:teuthology.orchestra.console:Hard reset for smithi205 completed
...
2020-04-06T08:16:11.025 DEBUG:teuthology.orchestra.remote:timed out
2020-04-06T08:16:11.025 DEBUG:teuthology.misc:waited 60.0049200058
2020-04-06T08:16:11.067 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 162, in invoke_logrotate
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
    self.ensure_online()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
    self.connect()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
    self.ssh = connection.connect(**args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 349, in connect
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/util.py", line 283, in retry_on_signal
    return function()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 349, in <lambda>
    retry_on_signal(lambda: sock.connect(addr))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/_socket2.py", line 249, in connect
    self._wait(self._write_event)
  File "src/gevent/_hub_primitives.py", line 284, in gevent.__hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 289, in gevent.__hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 280, in gevent.__hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 281, in gevent.__hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent.__waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
timeout: timed out

From: /ceph/teuthology-archive/teuthology-2020-04-06_04:15:02-multimds-master-testing-basic-smithi/4927617/teuthology.log

and

2020-05-06T14:15:42.200 INFO:teuthology.orchestra.console:Performing hard reset of smithi041
2020-05-06T14:15:42.201 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power reset
2020-05-06T14:15:42.230 INFO:teuthology.orchestra.console:Hard reset for smithi041 completed
...
2020-05-06T14:15:45.204 INFO:teuthology.orchestra.run.smithi041:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-05-06T14:15:45.253 INFO:teuthology.orchestra.run.smithi068:> true
2020-05-06T14:15:45.273 INFO:teuthology.orchestra.run.smithi068:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-05-06T14:15:45.318 INFO:teuthology.orchestra.run.smithi073:> true
2020-05-06T14:15:45.337 INFO:teuthology.orchestra.run.smithi073:> sudo logrotate /etc/logrotate.d/ceph-test.conf
...
2020-05-06T14:16:12.333 INFO:teuthology.misc:Re-opening connections...
2020-05-06T14:16:12.334 INFO:teuthology.misc:trying to connect to ubuntu@smithi041.front.sepia.ceph.com
2020-05-06T14:16:12.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-05-06T14:16:12.337 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi041.front.sepia.ceph.com', 'timeout': 60}
2020-05-06T14:16:12.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~43s
2020-05-06T14:16:15.450 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi041.front.sepia.ceph.com', 'timeout': 60}
2020-05-06T14:16:19.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~50s
2020-05-06T14:16:25.824 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-05-05-1439/qa/tasks/ceph.py", line 162, in invoke_logrotate
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 202, in run
    self.ensure_online()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 176, in ensure_online
    self.connect()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 72, in connect
    self.ssh = connection.connect(**args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/connection.py", line 108, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 368, in connect
    raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.41

From: /ceph/teuthology-archive/yuriw-2020-05-05_20:57:01-multimds-wip-yuri-testing-2020-05-05-1439-distro-basic-smithi/5026248/teuthology.log