Infrastructure

/a/yuriw-2023-05-22_23:22:00-rados-wip-yuri-testing-2023-05-22-0845-reef-distro-default-smithi/7282910

2023-05-22T23:52:21.434 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and
2023-05-22T23:52:21.434 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture.  It supports a rich
2023-05-22T23:52:21.434 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and
2023-05-22T23:52:21.435 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices.
2023-05-22T23:52:21.435 INFO:teuthology.orchestra.run.smithi187.stdout:
2023-05-22T23:52:21.435 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms.
2023-05-22T23:52:21.436 INFO:teuthology.task.kernel:Updating Grub Version: grub2
2023-05-22T23:52:21.437 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 4.18.0-490.el8.x86_64
2023-05-22T23:52:21.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg
2023-05-22T23:52:22.691 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ...
2023-05-22T23:52:24.548 INFO:teuthology.orchestra.run.smithi187.stderr:done
2023-05-22T23:52:24.549 DEBUG:teuthology.orchestra.run.smithi187:> mktemp
2023-05-22T23:52:24.601 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.LuH1VbbOQv
2023-05-22T23:52:24.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LuH1VbbOQv
2023-05-22T23:52:24.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.LuH1VbbOQv
2023-05-22T23:52:24.784 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.LuH1VbbOQv is 6KB
2023-05-22T23:52:24.838 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-05-22T23:52:24.840 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 130, in __init__
    server_version = self._send_version()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 134, in _send_version
    t, data = self._read_packet()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 201, in _read_packet
    x = self._read_all(4)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 188, in _read_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 1237, in task
    p.spawn(process_role, ctx, config, timeout, role, role_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 1341, in process_role
    install_and_reboot(ctx, {role: need_install}, config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 504, in install_and_reboot
    install_kernel(role_remote, config[role], version=src)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 857, in install_kernel
    update_grub_rpm(remote, version)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 922, in update_grub_rpm
    grub2_kernel_select_generic(remote, newversion, 'rpm')
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 940, in grub2_kernel_select_generic
    grub2conf = teuthology.get_file(remote, grubconfig, sudo=True).decode()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/misc.py", line 718, in get_file
    local_path = remote.get_file(path, sudo=sudo, dest_dir=dest_dir)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/orchestra/remote.py", line 620, in get_file
    self._sftp_get_file(path, local_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/orchestra/remote.py", line 545, in _sftp_get_file
    sftp = self.ssh.open_sftp()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 558, in open_sftp
    return self._transport.open_sftp_client()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1141, in open_sftp_client
    return SFTPClient.from_transport(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 170, in from_transport
    return cls(chan)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 132, in __init__
    raise SSHException("EOF during negotiation")
paramiko.ssh_exception.SSHException: EOF during negotiation
2023-05-22T23:52:24.928 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1bdd55c795c64ea99f71a06cd3c23852
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 130, in __init__
    server_version = self._send_version()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 134, in _send_version
    t, data = self._read_packet()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 201, in _read_packet
    x = self._read_all(4)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp.py", line 188, in _read_all
    raise EOFError()
EOFError
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/run_tasks.py", line 109, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 1237, in task
    p.spawn(process_role, ctx, config, timeout, role, role_config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 1341, in process_role
    install_and_reboot(ctx, {role: need_install}, config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 504, in install_and_reboot
    install_kernel(role_remote, config[role], version=src)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 857, in install_kernel
    update_grub_rpm(remote, version)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 922, in update_grub_rpm
    grub2_kernel_select_generic(remote, newversion, 'rpm')
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/task/kernel.py", line 940, in grub2_kernel_select_generic
    grub2conf = teuthology.get_file(remote, grubconfig, sudo=True).decode()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/misc.py", line 718, in get_file
    local_path = remote.get_file(path, sudo=sudo, dest_dir=dest_dir)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/orchestra/remote.py", line 620, in get_file
    self._sftp_get_file(path, local_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/teuthology/orchestra/remote.py", line 545, in _sftp_get_file
    sftp = self.ssh.open_sftp()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 558, in open_sftp
    return self._transport.open_sftp_client()
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1141, in open_sftp_client
    return SFTPClient.from_transport(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 170, in from_transport
    return cls(chan)
  File "/home/teuthworker/src/git.ceph.com_teuthology_170232cd589805f3a253a3fe352ccdf09a6a46bc/virtualenv/lib/python3.8/site-packages/paramiko/sftp_client.py", line 132, in __init__
    raise SSHException("EOF during negotiation")
paramiko.ssh_exception.SSHException: EOF during negotiation