Actions
Bug #14148
closedvps nodes lose connection after reboot
Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/hammer-x
Crash signature (v1):
Crash signature (v2):
Description
Seems to be on CentOS 7.0
Run: http://pulpito.ceph.com/teuthology-2015-12-19_08:57:51-upgrade:hammer-x-infernalis-distro-basic-vps/
Jobs: all on centos 7.0
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-12-19_08:57:51-upgrade:hammer-x-infernalis-distro-basic-vps/292/teuthology.log
2015-12-20T09:21:01.804 INFO:teuthology.orchestra.run.vpm026:Running: 'sudo shutdown -r now'
2015-12-20T09:21:01.808 INFO:teuthology.misc:Re-opening connections...
2015-12-20T09:21:01.809 INFO:teuthology.misc:trying to connect to ubuntu@vpm117.front.sepia.ceph.com
2015-12-20T09:21:01.809 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm117.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:22:01.814 DEBUG:teuthology.orchestra.remote:timed out
2015-12-20T09:22:01.815 INFO:teuthology.misc:trying to connect to ubuntu@vpm026.front.sepia.ceph.com
2015-12-20T09:22:01.815 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm026.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:23:01.819 DEBUG:teuthology.orchestra.remote:timed out
2015-12-20T09:23:01.819 DEBUG:teuthology.misc:waited 120.010210991
2015-12-20T09:23:02.820 INFO:teuthology.misc:trying to connect to ubuntu@vpm117.front.sepia.ceph.com
2015-12-20T09:23:02.821 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm117.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:24:02.824 DEBUG:teuthology.orchestra.remote:timed out
2015-12-20T09:24:02.825 INFO:teuthology.misc:trying to connect to ubuntu@vpm026.front.sepia.ceph.com
2015-12-20T09:24:02.825 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm026.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:24:05.825 DEBUG:teuthology.orchestra.remote:[Errno 113] No route to host
2015-12-20T09:24:05.825 DEBUG:teuthology.misc:waited 184.016594172
2015-12-20T09:24:06.826 INFO:teuthology.misc:trying to connect to ubuntu@vpm117.front.sepia.ceph.com
2015-12-20T09:24:06.826 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm117.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:25:06.831 DEBUG:teuthology.orchestra.remote:timed out
2015-12-20T09:25:06.831 INFO:teuthology.misc:trying to connect to ubuntu@vpm026.front.sepia.ceph.com
2015-12-20T09:25:06.831 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm026.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:25:09.833 DEBUG:teuthology.orchestra.remote:[Errno 113] No route to host
2015-12-20T09:25:09.833 DEBUG:teuthology.misc:waited 248.02457118
2015-12-20T09:25:10.834 INFO:teuthology.misc:trying to connect to ubuntu@vpm117.front.sepia.ceph.com
2015-12-20T09:25:10.835 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm117.front.sepia.ceph.com', 'timeout': 60}
2015-12-20T09:26:10.839 DEBUG:teuthology.orchestra.remote:timed out
2015-12-20T09:26:10.840 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
return fn(**kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 1247, in task
wait_for_reboot(ctx, need_version, timeout)
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 627, in wait_for_reboot
teuthology.reconnect(ctx, timeout)
File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 974, in reconnect
remote.name)
RuntimeError: Could not reconnect to ubuntu@vpm117.front.sepia.ceph.com
2015-12-20T09:26:10.865 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=96867ffd331b496f997d067ea73d06f1
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
return fn(**kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 1247, in task
wait_for_reboot(ctx, need_version, timeout)
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 627, in wait_for_reboot
teuthology.reconnect(ctx, timeout)
File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 974, in reconnect
remote.name)
RuntimeError: Could not reconnect to ubuntu@vpm117.front.sepia.ceph.com
Updated by Anonymous almost 8 years ago
Looking this over, this behavior seems to be due to a timeout being reached because the kernel did not reboot within a certain time (default is 300 seconds). I think it might be good to try this with a higher timeout value:
The yaml file that defines the kernel task should contain something like:
kernel:
timeout: 900
Actions