Bug #45592
OSError: [Errno 113] No route to host
Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-05-16T09:59:44.263 INFO:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-05-16T09:59:55.308 INFO:tasks.ceph.mds.d.smithi112.stdout:starting mds.d at 2020-05-16T10:15:14.402 ERROR:paramiko.transport:Socket exception: No route to host (113) 2020-05-16T10:15:14.407 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2020-05-16T10:15:14.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-05-16T10:15:14.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-05-16T10:15:14.412 DEBUG:tasks.ceph:Missed logrotate, SSHException 2020-05-16T10:15:14.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2020-05-16T10:15:14.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2020-05-16T10:15:17.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2020-05-16T10:15:17.475 INFO:tasks.cephfs_test_runner:test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize) ... ERROR 2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner:ERROR 2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner: 2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner:====================================================================== 2020-05-16T10:15:17.477 INFO:tasks.cephfs_test_runner:ERROR: test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize) 2020-05-16T10:15:17.477 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/test_failover.py", line 248, in test_up_less_than_max 2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner: self.shrink(2) 2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/test_failover.py", line 213, in shrink 2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner: status = self.fs.wait_for_daemons(timeout=60+grace*2) 2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 918, in wait_for_daemons 2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner: status = self.status() 2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 333, in status 2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner: return FSStatus(self.mon_manager) 2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 68, in __init__ 2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner: self.map = json.loads(self.mon.raw_cluster_cmd("fs", "dump", "--format=json")) 2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd 2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner: stdout=BytesIO(), 2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run 2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 468, in run 2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner: r.execute() 2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 102, in execute 2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner: self.client.exec_command(self.command) 2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 508, in exec_command 2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner: chan = self._transport.open_session(timeout=timeout) 2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 879, in open_session 2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner: timeout=timeout, 2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1006, in open_channel 2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner: raise e 2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2055, in run 2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner: ptype, m = self.packetizer.read_message() 2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 459, in read_message 2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner: header = self.read_all(self.__block_size_in, check_rekey=True) 2020-05-16T10:15:17.486 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 301, in read_all 2020-05-16T10:15:17.486 INFO:tasks.cephfs_test_runner: x = self.__socket.recv(n) 2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_socket3.py", line 378, in recv 2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner: return _socket.socket.recv(self._sock, *args) 2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner:OSError: [Errno 113] No route to host
From: /ceph/teuthology-archive/pdonnell-2020-05-16_06:07:05-fs-wip-pdonnell-testing-20200516.030215-distro-basic-smithi/5060351/teuthology.log
History
#1 Updated by Patrick Donnelly almost 4 years ago
- Description updated (diff)
#2 Updated by Kyrylo Shatskyy almost 4 years ago
Patrick, this looks like a bug in the test, it does not wait nodes are up after reboot or failover (or whatever)... is it reproducible only on your private branch?
#3 Updated by Patrick Donnelly almost 4 years ago
Kyrylo Shatskyy wrote:
Patrick, this looks like a bug in the test, it does not wait nodes are up after reboot or failover (or whatever)... is it reproducible only on your private branch?
There is no reboot in this test. I need to do more testing to see if this occurs more.
#4 Updated by Josh Durgin over 2 years ago
- Status changed from New to Can't reproduce
Patrick says this hasn't occurred again.