Ceph » Orchestrator

2022-02-10T06:34:53.016 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now
2022-02-10T06:34:53.290 INFO:teuthology.orchestra.run.smithi139.stderr:Found linux image: /boot/vmlinuz-4.15.0-159-generic
2022-02-10T06:34:53.303 INFO:teuthology.orchestra.run.smithi139.stderr:Found initrd image: /boot/initrd.img-4.15.0-159-generic
2022-02-10T06:34:53.858 INFO:teuthology.orchestra.run.smithi046.stderr:Sourcing file `/etc/default/grub'
2022-02-10T06:34:53.866 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ...
2022-02-10T06:34:54.075 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-4.15.0-167-generic
2022-02-10T06:34:54.087 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-4.15.0-167-generic
2022-02-10T06:34:54.169 INFO:teuthology.orchestra.run.smithi139.stderr:done
2022-02-10T06:34:54.171 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now
2022-02-10T06:34:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:Found linux image: /boot/vmlinuz-4.15.0-159-generic
2022-02-10T06:34:54.375 INFO:teuthology.orchestra.run.smithi046.stderr:Found initrd image: /boot/initrd.img-4.15.0-159-generic
2022-02-10T06:34:55.033 INFO:teuthology.orchestra.run.smithi046.stderr:done
2022-02-10T06:34:55.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now
2022-02-10T06:35:23.085 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version...
2022-02-10T06:35:23.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:35:23.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:35:24.174 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version...
2022-02-10T06:35:24.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-10T06:35:24.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:35:25.040 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version...
2022-02-10T06:35:25.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:35:25.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:35:25.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:35:41.643 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:35:42.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139
2022-02-10T06:35:55.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:35:55.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:35:55.056 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:36:11.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:36:11.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:12.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-10T06:36:12.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:14.730 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:36:15.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.139
2022-02-10T06:36:25.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:36:25.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:25.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:36:44.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:36:44.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:45.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-10T06:36:45.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:45.809 DEBUG:teuthology.orchestra.run.smithi139:> true
2022-02-10T06:36:46.102 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-10T06:36:46.336 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.15.0-167-generic"...
2022-02-10T06:36:46.336 DEBUG:teuthology.orchestra.run.smithi139:> uname -r
2022-02-10T06:36:46.343 INFO:teuthology.orchestra.run.smithi139.stdout:4.15.0-167-generic
2022-02-10T06:36:46.344 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-167-generic vs 4.15.0-167-generic
2022-02-10T06:36:46.345 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install
2022-02-10T06:36:46.345 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu
2022-02-10T06:36:47.347 INFO:teuthology.task.kernel:Enabling kdb on mon.c...
2022-02-10T06:36:47.347 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc
2022-02-10T06:36:47.380 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1
2022-02-10T06:36:47.410 DEBUG:teuthology.parallel:result is None
2022-02-10T06:36:47.819 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:36:55.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:36:55.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:36:58.123 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:37:17.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:37:17.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:37:20.971 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:37:28.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:37:28.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:37:50.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:37:50.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:38:09.355 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:38:28.127 DEBUG:teuthology.orchestra.remote:timed out
2022-02-10T06:38:39.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:38:39.421 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:38:42.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:38:58.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:38:58.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:39:01.195 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:39:12.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:39:12.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:39:15.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:39:31.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:39:31.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:39:34.250 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:39:45.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:39:45.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:39:48.683 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:40:04.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com'
2022-02-10T06:40:04.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:40:07.339 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46
2022-02-10T06:40:18.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:40:18.685 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:40:21.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:40:23.088 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-167-generic"...
2022-02-10T06:40:23.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com'
2022-02-10T06:40:23.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2022-02-10T06:40:24.811 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81
2022-02-10T06:40:24.811 ERROR:teuthology.task.kernel:Saw exception

or something similar happening on a lot of ubuntu 18.04 jobs on pacific

http://pulpito.ceph.com/adking-2022-02-12_16:39:52-orch:cephadm-wip-adk-testing-2022-02-12-0153-pacific-distro-basic-smithi/
http://pulpito.ceph.com/adking-2022-02-10_04:55:39-orch:cephadm-wip-adk2-testing-2022-02-09-2058-pacific-distro-basic-smithi/
http://pulpito.ceph.com/yuriw-2022-02-09_22:52:18-rados-wip-yuri5-testing-2022-02-09-1322-pacific-distro-default-smithi/