Actions
Bug #45511
openHuge ugly traceback when killing jobs
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-05-11T19:25:27.242 DEBUG:teuthology.exit:Got signal 15; running 2 handlers...
2020-05-11T19:25:27.244 DEBUG:teuthology.task.console_log:Killing console logger for smithi162
2020-05-11T19:25:27.245 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
2020-05-11T19:25:27.248 DEBUG:teuthology.run_tasks:Unwinding manager install
2020-05-11T19:25:27.255 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
2020-05-11T19:25:27.257 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits...
2020-05-11T19:25:27.258 INFO:teuthology.orchestra.run.smithi061:> true
2020-05-11T19:25:35.460 INFO:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
2020-05-11T19:25:35.463 INFO:teuthology.orchestra.run.smithi162:> true
2020-05-11T19:25:36.483 INFO:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits
2020-05-11T19:25:37.687 INFO:teuthology.orchestra.run.smithi162:> true
2020-05-11T19:25:37.696 INFO:teuthology.orchestra.run.smithi162:> sudo lsb_release -is
2020-05-11T19:25:37.757 INFO:teuthology.orchestra.run.smithi162.stdout:Ubuntu
2020-05-11T19:25:37.757 DEBUG:teuthology.misc:System to be installed: Ubuntu
2020-05-11T19:25:37.758 INFO:teuthology.orchestra.run.smithi061:> true
2020-05-11T19:25:37.759 INFO:teuthology.task.install.deb:Removing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on Debian system.
2020-05-11T19:25:37.759 INFO:teuthology.orchestra.run.smithi162:> true
2020-05-11T19:25:37.779 INFO:teuthology.orchestra.run.smithi162:> for d in ceph cephadm ceph-mds ceph-mgr ceph-common ceph-fuse ceph-test radosgw python3-rados python3-rgw python3-cephfs python3-rbd libcephfs2 libcephfs-dev librados2 librbd1 rbd-fuse ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done
2020-05-11T19:25:37.837 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists...
2020-05-11T19:25:37.969 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree...
2020-05-11T19:25:37.971 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information...
2020-05-11T19:25:38.092 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required:
2020-05-11T19:25:38.093 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon ceph-osd libboost-iostreams1.71.0
2020-05-11T19:25:38.093 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them.
2020-05-11T19:25:38.105 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be REMOVED:
2020-05-11T19:25:38.105 INFO:teuthology.orchestra.run.smithi162.stdout: ceph*
2020-05-11T19:33:09.796 DEBUG:teuthology.exit:Got signal 15; running 2 handlers...
2020-05-11T19:33:09.797 DEBUG:teuthology.task.console_log:Killing console logger for smithi162
2020-05-11T19:33:09.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:33:12.863 ERROR:teuthology.run_tasks:Manager failed: install
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 216, in install
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task
yield
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 218, in install
remove_packages(ctx, config, package_list)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 104, in remove_packages
system_type = teuthology.get_system_type(remote)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1198, in get_system_type
system_value = remote.sh('sudo lsb_release -is').strip()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 247, in sh
proc=self.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61
2020-05-11T19:33:12.864 DEBUG:teuthology.run_tasks:Unwinding manager clock
2020-05-11T19:33:12.876 INFO:teuthology.task.clock:Checking final clock skew...
2020-05-11T19:33:12.876 INFO:teuthology.orchestra.run.smithi162:> true
2020-05-11T19:41:23.550 ERROR:paramiko.transport:Socket exception: No route to host (113)
2020-05-11T19:44:20.348 DEBUG:teuthology.exit:Got signal 15; running 2 handlers...
2020-05-11T19:44:20.349 DEBUG:teuthology.task.console_log:Killing console logger for smithi162
2020-05-11T19:44:20.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:44:22.750 ERROR:teuthology.run_tasks:Manager failed: clock
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/clock.py", line 56, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/clock.py", line 66, in task
'true'
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.162
2020-05-11T19:44:22.752 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2020-05-11T19:44:22.769 INFO:teuthology.task.ansible:Skipping ansible cleanup...
2020-05-11T19:44:22.769 DEBUG:teuthology.run_tasks:Unwinding manager selinux
2020-05-11T19:44:22.796 DEBUG:teuthology.run_tasks:Unwinding manager pcp
2020-05-11T19:44:22.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer
2020-05-11T19:44:22.826 INFO:teuthology.task.internal:Duration was 34934.489409 seconds
2020-05-11T19:44:22.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog
2020-05-11T19:44:22.843 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring...
2020-05-11T19:44:22.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:44:25.918 ERROR:teuthology.run_tasks:Manager failed: internal.syslog
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/syslog.py", line 71, in syslog
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/syslog.py", line 89, in syslog
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61
2020-05-11T19:44:25.919 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo
2020-05-11T19:44:25.931 INFO:teuthology.task.internal:Restoring /etc/sudoers...
2020-05-11T19:44:25.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:44:28.990 ERROR:teuthology.run_tasks:Manager failed: internal.sudo
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 400, in sudo
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 405, in sudo
path=sudoers_file, ext=backup_ext
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61
2020-05-11T19:44:28.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
2020-05-11T19:44:29.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:44:32.062 ERROR:teuthology.run_tasks:Manager failed: internal.coredump
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 435, in coredump
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 448, in coredump
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61
2020-05-11T19:44:32.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2020-05-11T19:44:32.074 INFO:teuthology.task.internal:Transferring archived files...
2020-05-11T19:44:32.075 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi/5042933/remote/smithi162
2020-05-11T19:44:32.079 INFO:teuthology.orchestra.run.smithi162:> true
2020-05-11T19:48:44.894 ERROR:paramiko.transport:Socket exception: No route to host (113)
2020-05-11T19:48:44.896 DEBUG:teuthology.orchestra.run:got remote process result: None
2020-05-11T19:48:44.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:48:47.966 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 351, in archive
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 366, in archive
misc.pull_directory(rem, archive_dir, path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 766, in pull_directory
r = remote.get_tar_stream(remotedir, sudo=True)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 446, in get_tar_stream
return self.run(args=args, wait=False, stdout=run.PIPE)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.162
2020-05-11T19:48:47.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2020-05-11T19:48:47.982 INFO:teuthology.task.internal:Not uploading archives.
2020-05-11T19:48:47.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2020-05-11T19:48:48.006 INFO:teuthology.task.internal:Tidying up after the test...
2020-05-11T19:48:48.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-11T19:48:51.070 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 40, in base
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task
check_status=False,
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
reraise(*exc)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
if exit(*exc):
File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait
status = self._get_exitstatus()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus
status = self._stdout_buf.channel.recv_exit_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status
self.status_event.wait()
File "src/gevent/event.py", line 127, in gevent._event.Event.wait
File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 50, in base
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp>
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run
self.ensure_online()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online
self.connect()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect
self.ssh = connection.connect(**args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect
ssh.connect(**connect_args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61
2020-05-11T19:48:51.071 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2020-05-11T19:48:51.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2020-05-11T19:48:51.100 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303)
2020-05-11T19:48:51.100 INFO:teuthology.nuke:Checking targets against current locks
2020-05-11T19:48:51.162 CRITICAL:teuthology:Uncaught exception
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 389, in main
run_tasks(tasks=config['tasks'], ctx=fake_ctx)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 196, in run_tasks
raise SystemExit(1)
SystemExit: 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology", line 11, in <module>
load_entry_point('teuthology', 'console_scripts', 'teuthology')()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/run.py", line 34, in main
teuthology.run.main(args)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 392, in main
report_outcome(config, archive, fake_ctx.summary, fake_ctx)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 253, in report_outcome
nuke(fake_ctx, fake_ctx.lock)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/nuke/__init__.py", line 250, in nuke
if ctx.name not in lock['description']:
TypeError: argument of type 'NoneType' is not iterable
From http://pulpito.ceph.com/teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi/
Updated by David Galloway almost 4 years ago
Not sure if this is related?
dgalloway@teuthology:/a$ teuthology-kill -r teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi -j 5042869
2020-05-11 19:57:50,825.825 INFO:teuthology.kill:Killing Pids: {16683}
2020-05-11 19:57:50,844.844 INFO:teuthology.kill:Nuking machines: ['smithi002', 'smithi136']
2020-05-11 19:57:51,602.602 INFO:teuthology.kill:2020-05-11 19:57:51,601.601 INFO:teuthology.nuke:targets:
2020-05-11 19:57:51,602.602 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE=
2020-05-11 19:57:51,602.602 INFO:teuthology.kill: smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8=
2020-05-11 19:57:51,634.634 INFO:teuthology.kill:2020-05-11 19:57:51,634.634 INFO:teuthology.task.internal.check_lock:Checking locks...
2020-05-11 19:57:51,654.654 INFO:teuthology.kill:2020-05-11 19:57:51,654.654 ERROR:teuthology.nuke:Could not nuke {'smithi002.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE='}
2020-05-11 19:57:51,654.654 INFO:teuthology.kill:Traceback (most recent call last):
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 292, in nuke_one
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: nuke_helper(ctx, should_unlock)
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 315, in nuke_helper
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: check_lock.check_lock(ctx, None, check_up=False)
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/task/internal/check_lock.py", line 29, in check_lock
2020-05-11 19:57:51,655.655 INFO:teuthology.kill: 'machine {name} is not locked'.format(name=machine)
2020-05-11 19:57:51,655.655 INFO:teuthology.kill:AssertionError: machine smithi002.front.sepia.ceph.com is not locked
2020-05-11 19:57:51,656.656 INFO:teuthology.kill:2020-05-11 19:57:51,656.656 INFO:teuthology.task.internal.check_lock:Checking locks...
2020-05-11 19:57:51,672.672 INFO:teuthology.kill:2020-05-11 19:57:51,672.672 ERROR:teuthology.nuke:Could not nuke {'smithi136.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8='}
2020-05-11 19:57:51,672.672 INFO:teuthology.kill:Traceback (most recent call last):
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 292, in nuke_one
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: nuke_helper(ctx, should_unlock)
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 315, in nuke_helper
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: check_lock.check_lock(ctx, None, check_up=False)
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/task/internal/check_lock.py", line 29, in check_lock
2020-05-11 19:57:51,672.672 INFO:teuthology.kill: 'machine {name} is not locked'.format(name=machine)
2020-05-11 19:57:51,672.672 INFO:teuthology.kill:AssertionError: machine smithi136.front.sepia.ceph.com is not locked
2020-05-11 19:57:51,673.673 INFO:teuthology.kill:2020-05-11 19:57:51,673.673 ERROR:teuthology.nuke:Could not nuke the following targets:
2020-05-11 19:57:51,673.673 INFO:teuthology.kill:targets:
2020-05-11 19:57:51,673.673 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE=
2020-05-11 19:57:51,674.674 INFO:teuthology.kill: smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8=
Actions