Actions
Bug #45511
openHuge ugly traceback when killing jobs
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-05-11T19:25:27.242 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-05-11T19:25:27.244 DEBUG:teuthology.task.console_log:Killing console logger for smithi162 2020-05-11T19:25:27.245 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) 2020-05-11T19:25:27.248 DEBUG:teuthology.run_tasks:Unwinding manager install 2020-05-11T19:25:27.255 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) 2020-05-11T19:25:27.257 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2020-05-11T19:25:27.258 INFO:teuthology.orchestra.run.smithi061:> true 2020-05-11T19:25:35.460 INFO:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2020-05-11T19:25:35.463 INFO:teuthology.orchestra.run.smithi162:> true 2020-05-11T19:25:36.483 INFO:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2020-05-11T19:25:37.687 INFO:teuthology.orchestra.run.smithi162:> true 2020-05-11T19:25:37.696 INFO:teuthology.orchestra.run.smithi162:> sudo lsb_release -is 2020-05-11T19:25:37.757 INFO:teuthology.orchestra.run.smithi162.stdout:Ubuntu 2020-05-11T19:25:37.757 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-05-11T19:25:37.758 INFO:teuthology.orchestra.run.smithi061:> true 2020-05-11T19:25:37.759 INFO:teuthology.task.install.deb:Removing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on Debian system. 2020-05-11T19:25:37.759 INFO:teuthology.orchestra.run.smithi162:> true 2020-05-11T19:25:37.779 INFO:teuthology.orchestra.run.smithi162:> for d in ceph cephadm ceph-mds ceph-mgr ceph-common ceph-fuse ceph-test radosgw python3-rados python3-rgw python3-cephfs python3-rbd libcephfs2 libcephfs-dev librados2 librbd1 rbd-fuse ; do sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" purge $d || true ; done 2020-05-11T19:25:37.837 INFO:teuthology.orchestra.run.smithi162.stdout:Reading package lists... 2020-05-11T19:25:37.969 INFO:teuthology.orchestra.run.smithi162.stdout:Building dependency tree... 2020-05-11T19:25:37.971 INFO:teuthology.orchestra.run.smithi162.stdout:Reading state information... 2020-05-11T19:25:38.092 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages were automatically installed and are no longer required: 2020-05-11T19:25:38.093 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon ceph-osd libboost-iostreams1.71.0 2020-05-11T19:25:38.093 INFO:teuthology.orchestra.run.smithi162.stdout:Use 'sudo apt autoremove' to remove them. 2020-05-11T19:25:38.105 INFO:teuthology.orchestra.run.smithi162.stdout:The following packages will be REMOVED: 2020-05-11T19:25:38.105 INFO:teuthology.orchestra.run.smithi162.stdout: ceph* 2020-05-11T19:33:09.796 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-05-11T19:33:09.797 DEBUG:teuthology.task.console_log:Killing console logger for smithi162 2020-05-11T19:33:09.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:33:12.863 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 216, in install yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 599, in task yield File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 218, in install remove_packages(ctx, config, package_list) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 104, in remove_packages system_type = teuthology.get_system_type(remote) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 1198, in get_system_type system_value = remote.sh('sudo lsb_release -is').strip() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 247, in sh proc=self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61 2020-05-11T19:33:12.864 DEBUG:teuthology.run_tasks:Unwinding manager clock 2020-05-11T19:33:12.876 INFO:teuthology.task.clock:Checking final clock skew... 2020-05-11T19:33:12.876 INFO:teuthology.orchestra.run.smithi162:> true 2020-05-11T19:41:23.550 ERROR:paramiko.transport:Socket exception: No route to host (113) 2020-05-11T19:44:20.348 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-05-11T19:44:20.349 DEBUG:teuthology.task.console_log:Killing console logger for smithi162 2020-05-11T19:44:20.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:44:22.750 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/clock.py", line 56, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/clock.py", line 66, in task 'true' File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.162 2020-05-11T19:44:22.752 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2020-05-11T19:44:22.769 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2020-05-11T19:44:22.769 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2020-05-11T19:44:22.796 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2020-05-11T19:44:22.810 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2020-05-11T19:44:22.826 INFO:teuthology.task.internal:Duration was 34934.489409 seconds 2020-05-11T19:44:22.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2020-05-11T19:44:22.843 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2020-05-11T19:44:22.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:44:25.918 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/syslog.py", line 71, in syslog yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/syslog.py", line 89, in syslog wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp> return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61 2020-05-11T19:44:25.919 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2020-05-11T19:44:25.931 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2020-05-11T19:44:25.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:44:28.990 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 400, in sudo yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 405, in sudo path=sudoers_file, ext=backup_ext File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp> return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61 2020-05-11T19:44:28.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2020-05-11T19:44:29.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:44:32.062 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 435, in coredump yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 448, in coredump wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp> return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61 2020-05-11T19:44:32.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2020-05-11T19:44:32.074 INFO:teuthology.task.internal:Transferring archived files... 2020-05-11T19:44:32.075 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi/5042933/remote/smithi162 2020-05-11T19:44:32.079 INFO:teuthology.orchestra.run.smithi162:> true 2020-05-11T19:48:44.894 ERROR:paramiko.transport:Socket exception: No route to host (113) 2020-05-11T19:48:44.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2020-05-11T19:48:44.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:48:47.966 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 351, in archive yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 366, in archive misc.pull_directory(rem, archive_dir, path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 766, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 446, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.162 2020-05-11T19:48:47.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2020-05-11T19:48:47.982 INFO:teuthology.task.internal:Not uploading archives. 2020-05-11T19:48:47.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2020-05-11T19:48:48.006 INFO:teuthology.task.internal:Tidying up after the test... 2020-05-11T19:48:48.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-11T19:48:51.070 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 40, in base yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 1853, in task check_status=False, File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested reraise(*exc) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested if exit(*exc): File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph.py", line 268, in ceph_log wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 144, in wait status = self._get_exitstatus() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 193, in _get_exitstatus status = self._stdout_buf.channel.recv_exit_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 402, in recv_exit_status self.status_event.wait() File "src/gevent/event.py", line 127, in gevent._event.Event.wait File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 50, in base wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in <listcomp> return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 202, in run self.ensure_online() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 176, in ensure_online self.connect() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 72, in connect self.ssh = connection.connect(**args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 108, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.61 2020-05-11T19:48:51.071 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2020-05-11T19:48:51.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines 2020-05-11T19:48:51.100 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=5303) 2020-05-11T19:48:51.100 INFO:teuthology.nuke:Checking targets against current locks 2020-05-11T19:48:51.162 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 389, in main run_tasks(tasks=config['tasks'], ctx=fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 196, in run_tasks raise SystemExit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology", line 11, in <module> load_entry_point('teuthology', 'console_scripts', 'teuthology')() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/scripts/run.py", line 34, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 392, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run.py", line 253, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable
From http://pulpito.ceph.com/teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi/
Updated by David Galloway almost 4 years ago
Not sure if this is related?
dgalloway@teuthology:/a$ teuthology-kill -r teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi -j 5042869 2020-05-11 19:57:50,825.825 INFO:teuthology.kill:Killing Pids: {16683} 2020-05-11 19:57:50,844.844 INFO:teuthology.kill:Nuking machines: ['smithi002', 'smithi136'] 2020-05-11 19:57:51,602.602 INFO:teuthology.kill:2020-05-11 19:57:51,601.601 INFO:teuthology.nuke:targets: 2020-05-11 19:57:51,602.602 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE= 2020-05-11 19:57:51,602.602 INFO:teuthology.kill: smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8= 2020-05-11 19:57:51,634.634 INFO:teuthology.kill:2020-05-11 19:57:51,634.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-05-11 19:57:51,654.654 INFO:teuthology.kill:2020-05-11 19:57:51,654.654 ERROR:teuthology.nuke:Could not nuke {'smithi002.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE='} 2020-05-11 19:57:51,654.654 INFO:teuthology.kill:Traceback (most recent call last): 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 292, in nuke_one 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: nuke_helper(ctx, should_unlock) 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 315, in nuke_helper 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: check_lock.check_lock(ctx, None, check_up=False) 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/task/internal/check_lock.py", line 29, in check_lock 2020-05-11 19:57:51,655.655 INFO:teuthology.kill: 'machine {name} is not locked'.format(name=machine) 2020-05-11 19:57:51,655.655 INFO:teuthology.kill:AssertionError: machine smithi002.front.sepia.ceph.com is not locked 2020-05-11 19:57:51,656.656 INFO:teuthology.kill:2020-05-11 19:57:51,656.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-05-11 19:57:51,672.672 INFO:teuthology.kill:2020-05-11 19:57:51,672.672 ERROR:teuthology.nuke:Could not nuke {'smithi136.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8='} 2020-05-11 19:57:51,672.672 INFO:teuthology.kill:Traceback (most recent call last): 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 292, in nuke_one 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: nuke_helper(ctx, should_unlock) 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/nuke/__init__.py", line 315, in nuke_helper 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: check_lock.check_lock(ctx, None, check_up=False) 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: File "/home/dgalloway/src/teuthology/teuthology/task/internal/check_lock.py", line 29, in check_lock 2020-05-11 19:57:51,672.672 INFO:teuthology.kill: 'machine {name} is not locked'.format(name=machine) 2020-05-11 19:57:51,672.672 INFO:teuthology.kill:AssertionError: machine smithi136.front.sepia.ceph.com is not locked 2020-05-11 19:57:51,673.673 INFO:teuthology.kill:2020-05-11 19:57:51,673.673 ERROR:teuthology.nuke:Could not nuke the following targets: 2020-05-11 19:57:51,673.673 INFO:teuthology.kill:targets: 2020-05-11 19:57:51,673.673 INFO:teuthology.kill: smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR1+zHygI7G2tR4fdWOEyisLfB3cMEYq78/ZbHhnq1Q5KIZMxLpxa9ncHPD0nkhGOIC3yaengjc28vWWx0dH7AmMywGN74Pn2hlTa7RmOc6XZdUElHWrA18oTBTRsuyecCf7rsloBkC7ZEgmYrq4v1BRsacTfeQuscL3B0GuKxskX4f1sXgoWNu59qhPnJlr4qWZk6NYBpRV+YeTk3bpC+K2SpRALgJWBQDLuOemNurXeagUxAvsdxXRaVxm9QFgNLPzyq8eSY1XG1Y7GOd8pxoYFh7jEOtbO6fZa5AGPjPMFG8FeT+XCHQ71qCm5UgC7KdUw0OitfF5TAE69SR8/65DkH/6PAkOgRweQdlE6YGkunbdvmfONu8xvVzpxCSFaPA84MeWxgJ+blLgm/ClgJePDUME3EXGXsiXKLW1xZyHuu0TmFHzS7RmV9drfpaUNur2g9Src1WJQw6q4+9AhOH0StePgDpt6ksG3yEdpefd1cal+PY6pvBQ/9dF9FrdE= 2020-05-11 19:57:51,674.674 INFO:teuthology.kill: smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7JtDFtUMO863zWCdPDyv3qJZty3f7dieECl/It3tEQFwCnglxnlnwUetx3SI0yq5QwV90C8kSz+FnO8rsysojAluDSojdzbdtOdBINbDbYlIKzb6LutPOqDTdJXD5KKc9RON90MqlDhx2mLroikJ2gfuolJsoH645svtmV9JV6kh5cUs4uWjMb2EOe6y6sJtP7ZF27KWz5oHdYNv6wPTNkl720aX43cOWWmQFKHSfRJX91hSnnTEqQvKbgmI3Mk9xoR0tdHR3CMT67ZPUzfUTzBVJzugPagM2MOaLZBBY6GhWPCakuF565/u1r3MHCeEis8zkdr55jKvHOEAnVL+jojHBSZlsvkyc3di8NJRQ1uZJbN4GcTkIpZwW6MuT0NMrFc10zGpIw4PZSH/fEIVWOC9+8SMX1w2GUjuv1WRlp++7sOkSAl7TP5m5QDGSDYMLQKFPzHPA7KEECzZS5Gl7zKaqS6lM+I8zeJ4EmwU9wv8qilc5qfDdDc7JkBQLXk8=
Actions