Project

General

Profile

Bug #45592

OSError: [Errno 113] No route to host

Added by Patrick Donnelly over 1 year ago. Updated 3 months ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2020-05-16T09:59:44.263 INFO:teuthology.orchestra.run.smithi019:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-05-16T09:59:55.308 INFO:tasks.ceph.mds.d.smithi112.stdout:starting mds.d at
2020-05-16T10:15:14.402 ERROR:paramiko.transport:Socket exception: No route to host (113)
2020-05-16T10:15:14.407 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules...
2020-05-16T10:15:14.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-05-16T10:15:14.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-05-16T10:15:14.412 DEBUG:tasks.ceph:Missed logrotate, SSHException
2020-05-16T10:15:14.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s
2020-05-16T10:15:14.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s
2020-05-16T10:15:17.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19
2020-05-16T10:15:17.475 INFO:tasks.cephfs_test_runner:test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize) ... ERROR
2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner:ERROR
2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner:
2020-05-16T10:15:17.476 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-16T10:15:17.477 INFO:tasks.cephfs_test_runner:ERROR: test_up_less_than_max (tasks.cephfs.test_failover.TestClusterResize)
2020-05-16T10:15:17.477 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/test_failover.py", line 248, in test_up_less_than_max
2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner:    self.shrink(2)
2020-05-16T10:15:17.478 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/test_failover.py", line 213, in shrink
2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner:    status = self.fs.wait_for_daemons(timeout=60+grace*2)
2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 918, in wait_for_daemons
2020-05-16T10:15:17.479 INFO:tasks.cephfs_test_runner:    status = self.status()
2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 333, in status
2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner:    return FSStatus(self.mon_manager)
2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/cephfs/filesystem.py", line 68, in __init__
2020-05-16T10:15:17.480 INFO:tasks.cephfs_test_runner:    self.map = json.loads(self.mon.raw_cluster_cmd("fs", "dump", "--format=json"))
2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20200516.030215/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-05-16T10:15:17.481 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 468, in run
2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner:    r.execute()
2020-05-16T10:15:17.482 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 102, in execute
2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner:    self.client.exec_command(self.command)
2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/client.py", line 508, in exec_command
2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner:    chan = self._transport.open_session(timeout=timeout)
2020-05-16T10:15:17.483 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 879, in open_session
2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner:    timeout=timeout,
2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 1006, in open_channel
2020-05-16T10:15:17.484 INFO:tasks.cephfs_test_runner:    raise e
2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2055, in run
2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner:    ptype, m = self.packetizer.read_message()
2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 459, in read_message
2020-05-16T10:15:17.485 INFO:tasks.cephfs_test_runner:    header = self.read_all(self.__block_size_in, check_rekey=True)
2020-05-16T10:15:17.486 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 301, in read_all
2020-05-16T10:15:17.486 INFO:tasks.cephfs_test_runner:    x = self.__socket.recv(n)
2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_socket3.py", line 378, in recv
2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner:    return _socket.socket.recv(self._sock, *args)
2020-05-16T10:15:17.487 INFO:tasks.cephfs_test_runner:OSError: [Errno 113] No route to host

From: /ceph/teuthology-archive/pdonnell-2020-05-16_06:07:05-fs-wip-pdonnell-testing-20200516.030215-distro-basic-smithi/5060351/teuthology.log

History

#1 Updated by Patrick Donnelly over 1 year ago

  • Description updated (diff)

#2 Updated by Kyrylo Shatskyy over 1 year ago

Patrick, this looks like a bug in the test, it does not wait nodes are up after reboot or failover (or whatever)... is it reproducible only on your private branch?

#3 Updated by Patrick Donnelly over 1 year ago

Kyrylo Shatskyy wrote:

Patrick, this looks like a bug in the test, it does not wait nodes are up after reboot or failover (or whatever)... is it reproducible only on your private branch?

There is no reboot in this test. I need to do more testing to see if this occurs more.

#4 Updated by Josh Durgin 3 months ago

  • Status changed from New to Can't reproduce

Patrick says this hasn't occurred again.

Also available in: Atom PDF