Project

General

Profile

Actions
Copy link

Bug #45527

closed

mgr/dashboard: ConnectionLostError in "setUpClass (tasks.mgr.dashboard.test_host.HostControllerNoOrchestratorTest)"

Added by Laura Paduano almost 4 years ago. Updated about 3 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
Testing & QA
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seems to only happen in Ubuntu 20.04 nodes.

2020-05-12T15:29:02.793 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  1 -- 172.21.15.22:0/1828902899 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd000cdc0 con 0x7effdc0a5710
2020-05-12T15:29:02.794 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effdb59e700  1 -- 172.21.15.22:0/1828902899 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc148df0 con 0x7effdc0a5710
2020-05-12T15:29:02.794 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  1 -- 172.21.15.22:0/1828902899 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effd0012280 con 0x7effdc0a5710
2020-05-12T15:29:02.794 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effdb59e700  1 -- 172.21.15.22:0/1828902899 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc13db20 con 0x7effdc0a5710
2020-05-12T15:29:02.794 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effda59c700  1 --2- 172.21.15.22:0/1828902899 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7effdc0a5710 0x7effdc062d00 secure :-1 s=THROTTLE_DONE pgs=2072 cs=0 l=1 rx=0x7effdc05fe80 tx=0x7effd00013f0).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-05-12T15:29:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  1 -- 172.21.15.22:0/1828902899 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 165) v1 ==== 30785+0+0 (secure 0 0 0) 0x7effd0006520 con 0x7effdc0a5710
2020-05-12T15:29:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700 20 mgrc handle_mgr_map mgrmap(e 165) v1
2020-05-12T15:29:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effda59c700  1 --2- 172.21.15.22:0/1828902899 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7effdc0a5710 0x7effdc062d00 secure :-1 s=THROTTLE_DONE pgs=2072 cs=0 l=1 rx=0x7effdc05fe80 tx=0x7effd00013f0).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-05-12T15:29:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  4 mgrc handle_mgr_map Got map version 165
2020-05-12T15:29:02.795 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  4 mgrc handle_mgr_map Active mgr is now
2020-05-12T15:29:02.796 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  4 mgrc reconnect No active mgr available yet
2020-05-12T15:29:02.796 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effcb7fe700  1 -- 172.21.15.22:0/1828902899 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 13745+0+0 (secure 0 0 0) 0x7effd0035be0 con 0x7effdc0a5710
2020-05-12T15:29:02.796 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.768+0000 7effdb59e700  1 -- 172.21.15.22:0/1828902899 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc0569e0 con 0x7effdc0a5710
2020-05-12T15:29:02.797 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.776+0000 7effda59c700  1 --2- 172.21.15.22:0/1828902899 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7effdc0a5710 0x7effdc062d00 secure :-1 s=THROTTLE_DONE pgs=2072 cs=0 l=1 rx=0x7effdc05fe80 tx=0x7effd00013f0).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-05-12T15:29:02.797 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.776+0000 7effcb7fe700  1 -- 172.21.15.22:0/1828902899 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+129441 (secure 0 0 0) 0x7effd000fdb0 con 0x7effdc0a5710
2020-05-12T15:29:02.934 INFO:teuthology.orchestra.run.smithi022.stderr:2020-05-12T15:29:02.928+0000 7effdb59e700  1 -- 172.21.15.22:0/1828902899 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "dashboard"} v 0) v1 -- 0x7effdc06aa10 con 0x7effdc0a5710
2020-05-12T15:29:31.937 INFO:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2020-05-12T15:45:06.782 ERROR:paramiko.transport:Socket exception: No route to host (113)
2020-05-12T15:45:06.787 DEBUG:teuthology.orchestra.run:got remote process result: None
2020-05-12T15:45:06.789 INFO:tasks.cephfs_test_runner:ERROR
2020-05-12T15:45:06.790 INFO:tasks.cephfs_test_runner:
2020-05-12T15:45:06.790 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-12T15:45:06.790 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_host.HostControllerNoOrchestratorTest)
2020-05-12T15:45:06.791 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:45:06.791 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-12T15:45:06.791 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/dashboard/helper.py", line 143, in setUpClass
2020-05-12T15:45:06.791 INFO:tasks.cephfs_test_runner:    super(DashboardTestCase, cls).setUpClass()
2020-05-12T15:45:06.792 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/mgr_test_case.py", line 106, in setUpClass
2020-05-12T15:45:06.792 INFO:tasks.cephfs_test_runner:    cls.setup_mgrs()
2020-05-12T15:45:06.792 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/mgr_test_case.py", line 80, in setup_mgrs
2020-05-12T15:45:06.792 INFO:tasks.cephfs_test_runner:    "mgr", "module", "disable", m)
2020-05-12T15:45:06.793 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-05-12T15:45:06.793 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-05-12T15:45:06.793 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-05-12T15:45:06.794 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-05-12T15:45:06.794 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-05-12T15:45:06.794 INFO:tasks.cephfs_test_runner:    r.wait()
2020-05-12T15:45:06.794 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-05-12T15:45:06.795 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-05-12T15:45:06.795 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status
2020-05-12T15:45:06.795 INFO:tasks.cephfs_test_runner:    node=self.hostname)
2020-05-12T15:45:06.795 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi022 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr module disable dashboard'
2020-05-12T15:45:06.796 INFO:tasks.cephfs_test_runner:
2020-05-12T15:45:06.796 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:45:06.796 INFO:tasks.cephfs_test_runner:Ran 71 tests in 1804.586s
2020-05-12T15:45:06.797 INFO:tasks.cephfs_test_runner:
2020-05-12T15:45:06.797 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2020-05-12T15:45:06.797 INFO:tasks.cephfs_test_runner:
2020-05-12T15:45:06.797 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-12T15:45:06.808 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_host.HostControllerNoOrchestratorTest)
2020-05-12T15:45:06.808 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:45:06.808 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-12T15:45:06.808 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/dashboard/helper.py", line 143, in setUpClass
2020-05-12T15:45:06.809 INFO:tasks.cephfs_test_runner:    super(DashboardTestCase, cls).setUpClass()
2020-05-12T15:45:06.809 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/mgr_test_case.py", line 106, in setUpClass
2020-05-12T15:45:06.809 INFO:tasks.cephfs_test_runner:    cls.setup_mgrs()
2020-05-12T15:45:06.810 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/mgr/mgr_test_case.py", line 80, in setup_mgrs
2020-05-12T15:45:06.810 INFO:tasks.cephfs_test_runner:    "mgr", "module", "disable", m)
2020-05-12T15:45:06.810 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-05-12T15:45:06.810 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-05-12T15:45:06.811 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-05-12T15:45:06.811 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-05-12T15:45:06.811 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-05-12T15:45:06.811 INFO:tasks.cephfs_test_runner:    r.wait()
2020-05-12T15:45:06.812 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-05-12T15:45:06.812 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-05-12T15:45:06.812 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status
2020-05-12T15:45:06.812 INFO:tasks.cephfs_test_runner:    node=self.hostname)
2020-05-12T15:45:06.813 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi022 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early mgr module disable dashboard'
2020-05-12T15:45:06.813 INFO:tasks.cephfs_test_runner:
2020-05-12T15:45:06.813 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerNoOrchestratorTest)
2020-05-12T15:45:06.853 DEBUG:tasks.ceph:Missed logrotate, host unreachable
2020-05-12T15:45:06.870 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=e156acf5366d480782b935e418de57ac
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerNoOrchestratorTest)
2020-05-12T15:45:06.871 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2020-05-12T15:45:06.896 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2020-05-12T15:45:06.910 DEBUG:tasks.rgw.client.0:waiting for process to exit
2020-05-12T15:45:06.910 INFO:teuthology.orchestra.run:waiting for 300
2020-05-12T15:45:06.913 DEBUG:teuthology.orchestra.run:got remote process result: None
2020-05-12T15:45:06.913 ERROR:teuthology.run_tasks:Manager failed: rgw
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 167, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/rgw.py", line 391, in task
    yield
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 56, in nested
    reraise(*exc)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 48, in nested
    if exit(*exc):
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-34863/qa/tasks/rgw.py", line 190, in start_rgw
    ctx.daemons.get_daemon('rgw', client_with_id, cluster_name).stop()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/daemon/state.py", line 139, in stop
    run.wait([self.proc], timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 497, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status
    node=self.hostname)
teuthology.exceptions.ConnectionLostError: SSH connection to smithi022 was lost: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1" 
2020-05-12T15:45:06.914 DEBUG:teuthology.run_tasks:Unwinding manager ceph


2020-05-12T15:41:25.098 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-12T15:41:25.098 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_health.HealthTest)
2020-05-12T15:41:25.099 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:41:25.099 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-12T15:41:25.099 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/mgr/dashboard/helper.py", line 158, in setUpClass
2020-05-12T15:41:25.099 INFO:tasks.cephfs_test_runner:    cls.fs = cls.mds_cluster.newfs(create=True)
2020-05-12T15:41:25.100 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 330, in newfs
2020-05-12T15:41:25.100 INFO:tasks.cephfs_test_runner:    return Filesystem(self._ctx, name=name, create=create)
2020-05-12T15:41:25.100 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 448, in __init__
2020-05-12T15:41:25.101 INFO:tasks.cephfs_test_runner:    self.create()
2020-05-12T15:41:25.101 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 600, in create
2020-05-12T15:41:25.101 INFO:tasks.cephfs_test_runner:    self.mon_manager.raw_cluster_cmd('fs', 'set', self.name, 'standby_count_wanted', '0')
2020-05-12T15:41:25.101 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-05-12T15:41:25.105 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-05-12T15:41:25.105 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-05-12T15:41:25.106 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-05-12T15:41:25.106 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-05-12T15:41:25.106 INFO:tasks.cephfs_test_runner:    r.wait()
2020-05-12T15:41:25.106 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-05-12T15:41:25.107 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-05-12T15:41:25.107 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status
2020-05-12T15:41:25.107 INFO:tasks.cephfs_test_runner:    node=self.hostname)
2020-05-12T15:41:25.107 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi014 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early fs set cephfs standby_count_wanted 0'
2020-05-12T15:41:25.108 INFO:tasks.cephfs_test_runner:
2020-05-12T15:41:25.108 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:41:25.108 INFO:tasks.cephfs_test_runner:Ran 68 tests in 1802.024s
2020-05-12T15:41:25.109 INFO:tasks.cephfs_test_runner:
2020-05-12T15:41:25.109 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2020-05-12T15:41:25.109 INFO:tasks.cephfs_test_runner:
2020-05-12T15:41:25.110 INFO:tasks.cephfs_test_runner:======================================================================
2020-05-12T15:41:25.110 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_health.HealthTest)
2020-05-12T15:41:25.111 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-05-12T15:41:25.111 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-05-12T15:41:25.111 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/mgr/dashboard/helper.py", line 158, in setUpClass
2020-05-12T15:41:25.112 INFO:tasks.cephfs_test_runner:    cls.fs = cls.mds_cluster.newfs(create=True)
2020-05-12T15:41:25.112 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 330, in newfs
2020-05-12T15:41:25.112 INFO:tasks.cephfs_test_runner:    return Filesystem(self._ctx, name=name, create=create)
2020-05-12T15:41:25.112 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 448, in __init__
2020-05-12T15:41:25.113 INFO:tasks.cephfs_test_runner:    self.create()
2020-05-12T15:41:25.113 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/cephfs/filesystem.py", line 600, in create
2020-05-12T15:41:25.113 INFO:tasks.cephfs_test_runner:    self.mon_manager.raw_cluster_cmd('fs', 'set', self.name, 'standby_count_wanted', '0')
2020-05-12T15:41:25.113 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33121-3/qa/tasks/ceph_manager.py", line 1363, in raw_cluster_cmd
2020-05-12T15:41:25.114 INFO:tasks.cephfs_test_runner:    stdout=BytesIO(),
2020-05-12T15:41:25.114 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
2020-05-12T15:41:25.114 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-05-12T15:41:25.115 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-05-12T15:41:25.115 INFO:tasks.cephfs_test_runner:    r.wait()
2020-05-12T15:41:25.115 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-05-12T15:41:25.115 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-05-12T15:41:25.116 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 176, in _raise_for_status
2020-05-12T15:41:25.116 INFO:tasks.cephfs_test_runner:    node=self.hostname)
2020-05-12T15:41:25.116 INFO:tasks.cephfs_test_runner:teuthology.exceptions.ConnectionLostError: SSH connection to smithi014 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early fs set cephfs standby_count_wanted 0'

- http://pulpito.ceph.com/laura-2020-05-12_14:24:16-rados:dashboard-wip-laura-testing-33121-3-distro-basic-smithi/5049212/
- http://pulpito.ceph.com/laura-2020-05-12_14:26:26-rados:dashboard-wip-laura-testing-34863-distro-basic-smithi/

Actions
Copy link
 #1

Updated by Laura Paduano almost 4 years ago

  • Affected Versions v16.0.0 added
Actions
Copy link
 #2

Updated by Lenz Grimmer almost 4 years ago

This looks like an infrastructure issue to me?

2020-05-12T15:45:06.782 ERROR:paramiko.transport:Socket exception: No route to host (113)

and:
teuthology.exceptions.ConnectionLostError: SSH connection to smithi022 was lost

Actions
Copy link
 #3

Updated by Laura Paduano almost 4 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Laura Paduano almost 4 years ago

Lenz Grimmer wrote:

This looks like an infrastructure issue to me?
[...]
and:
[...]

Yes, and it only seems to affect Ubuntu 20.04 nodes..

Actions
Copy link
 #5

Updated by Laura Paduano almost 4 years ago

After chatting with Kefu, this tracker issue might be related:
https://tracker.ceph.com/issues/45438
And there's an open PR which might addresses the issue (but still, there are two different issues "Unable to connect to port" and "No route to host"):
https://github.com/ceph/teuthology/pull/1466

Actions
Copy link
 #6

Updated by Laura Paduano almost 4 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link
 #7

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 151 to Testing & QA
Actions
Copy link

Also available in: Atom PDF