Project

General

Profile

Actions
Copy link

Bug #19918

closed

xenial kernel update timing out or taking more time - unable to connect to port 22 on 172.21.2.91

Added by Vasu Kulkarni almost 7 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
High
Assignee:
Ilya Dryomov
Category:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

1) we also need to add some sleep before reconnect to avoid immediate reconnect ( should be fixed in teuthology )


2017-05-12T05:52:01.414 INFO:teuthology.orchestra.run.vpm091:Running: '/bin/echo -e \'cat <<EOF\\nset default="Advanced options for Ubuntu>Ubuntu, with Linux 4.11.0-ceph-g6f58448462f5"\\nEOF\\n\' | sudo tee -- \'/etc/grub.d/01_ceph_kernel.tmp~\' >/dev/null && sudo chmod a+x -- \'/etc/grub.d/01_ceph_kernel.tmp~\' && sudo mv -- \'/etc/grub.d/01_ceph_kernel.tmp~\' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & )'
2017-05-12T05:52:01.482 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@vpm091.front.sepia.ceph.com to complete...
2017-05-12T05:52:01.630 INFO:teuthology.orchestra.run.vpm091.stderr:Generating grub configuration file ...
2017-05-12T05:52:01.689 INFO:teuthology.orchestra.run.vpm091.stderr:Found linux image: /boot/vmlinuz-4.11.0-ceph-g6f58448462f5
2017-05-12T05:52:01.695 INFO:teuthology.orchestra.run.vpm091.stderr:Found initrd image: /boot/initrd.img-4.11.0-ceph-g6f58448462f5
2017-05-12T05:52:01.808 INFO:teuthology.orchestra.run.vpm091.stderr:Found linux image: /boot/vmlinuz-4.4.0-24-generic
2017-05-12T05:52:01.815 INFO:teuthology.orchestra.run.vpm091.stderr:Found initrd image: /boot/initrd.img-4.4.0-24-generic
2017-05-12T05:52:01.928 INFO:teuthology.orchestra.run.vpm091.stderr:done
2017-05-12T05:52:02.983 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@vpm165.front.sepia.ceph.com to complete...
2017-05-12T05:52:03.291 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@vpm131.front.sepia.ceph.com to complete...
2017-05-12T05:52:03.291 INFO:teuthology.misc:Re-opening connections...
2017-05-12T05:52:03.291 INFO:teuthology.misc:trying to connect to ubuntu@vpm165.front.sepia.ceph.com
2017-05-12T05:52:03.292 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm165.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:03.569 INFO:teuthology.orchestra.run.vpm165:Running: 'true'
2017-05-12T05:52:03.882 INFO:teuthology.misc:trying to connect to ubuntu@vpm091.front.sepia.ceph.com
2017-05-12T05:52:03.883 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm091.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:03.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.2.91
2017-05-12T05:52:03.884 DEBUG:teuthology.misc:waited 0.592638969421
2017-05-12T05:52:04.885 INFO:teuthology.misc:trying to connect to ubuntu@vpm131.front.sepia.ceph.com
2017-05-12T05:52:04.885 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm131.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:04.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.2.131
2017-05-12T05:52:04.887 INFO:teuthology.misc:trying to connect to ubuntu@vpm091.front.sepia.ceph.com
2017-05-12T05:52:04.887 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm091.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:04.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.2.91
2017-05-12T05:52:04.888 DEBUG:teuthology.misc:waited 1.5967900753
2017-05-12T05:52:05.889 INFO:teuthology.misc:trying to connect to ubuntu@vpm131.front.sepia.ceph.com
2017-05-12T05:52:05.890 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm131.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:05.891 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.2.131
2017-05-12T05:52:05.891 INFO:teuthology.misc:trying to connect to ubuntu@vpm091.front.sepia.ceph.com
2017-05-12T05:52:05.891 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm091.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
2017-05-12T05:52:21.255 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.2.91
2017-05-12T05:52:21.255 DEBUG:teuthology.misc:waited 17.9639761448
2017-05-12T05:52:22.257 INFO:teuthology.misc:trying to connect to ubuntu@vpm131.front.sepia.ceph.com
2017-05-12T05:52:22.258 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'vpm131.front.sepia.ceph.com', 'key_filename': ['/home/teuthworker/.ssh/id_rsa'], 'timeout': 60}
Actions
Copy link
 #2

Updated by Vasu Kulkarni almost 7 years ago

traceback:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 1315, in task
    wait_for_reboot(ctx, need_version, timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 692, in wait_for_reboot
    teuthology.reconnect(ctx, timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 999, in reconnect
    remote.name)
RuntimeError: Could not reconnect to ubuntu@vpm131.front.sepia.ceph.com
2017-05-12T05:57:03.738 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4f8c8eefdd474d28858bf7a3699cb491
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 1315, in task
    wait_for_reboot(ctx, need_version, timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 692, in wait_for_reboot
    teuthology.reconnect(ctx, timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/misc.py", line 999, in reconnect
    remote.name)
RuntimeError: Could not reconnect to ubuntu@vpm131.front.sepia.ceph.com
Actions
Copy link
 #3

Updated by Ilya Dryomov almost 7 years ago

  • Project changed from Ceph to teuthology
  • Priority changed from High to Normal

The rest of the log looks normal to me, but then I don't use vpses...

It looks like it occurs at least once on every other smoke run. I'll see if I can catch it in action as time permits.

Actions
Copy link
 #4

Updated by Vasu Kulkarni almost 7 years ago

  • Priority changed from Normal to High

Ilya any update on this? I have tried with delay of 60 seconds after kernel update(during reconnect) but that didn't help much.

http://pulpito.ceph.com/vasu-2017-07-20_01:02:30-smoke-master-testing-basic-vps/

Actions
Copy link
 #5

Updated by Ilya Dryomov almost 7 years ago

No, I'll try to take look this week.

Actions
Copy link
 #6

Updated by Ilya Dryomov about 5 years ago

  • Status changed from New to Closed

We aren't running suites on vpses anymore.

Actions
Copy link

Also available in: Atom PDF