Project

General

Profile

Bug #43335

AssertionError: failed to install new kernel version within timeout

Added by Patrick Donnelly about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2019-12-15T03:17:24.232 DEBUG:teuthology.task.kernel:get_image_version: 5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:24.232 INFO:teuthology.orchestra.run.smithi067:> rpm --scripts -qp /tmp/kernel.x86_64.rpm
2019-12-15T03:17:24.374 INFO:teuthology.orchestra.run.smithi067.stdout:postinstall scriptlet (using /bin/sh):
2019-12-15T03:17:24.374 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c -a -r /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c ]; then
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:cp /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:cp /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:rm -f /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/installkernel 5.5.0-rc1-ceph-g7943cf18163c /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:rm -f /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:fi
2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:preuninstall scriptlet (using /bin/sh):
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/new-kernel-pkg ]; then
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:new-kernel-pkg --remove 5.5.0-rc1-ceph-g7943cf18163c --rminitrd --initrdfile=/boot/initramfs-5.5.0-rc1-ceph-g7943cf18163c.img
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:elif [ -x /usr/bin/kernel-install ]; then
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-install remove 5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:fi
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:postuninstall scriptlet (using /bin/sh):
2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/update-bootloader ]; then
2019-12-15T03:17:24.377 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/update-bootloader --remove 5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:24.377 INFO:teuthology.orchestra.run.smithi067.stdout:fi
2019-12-15T03:17:24.378 INFO:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools
2019-12-15T03:17:24.563 INFO:teuthology.orchestra.run.smithi067.stdout:Name        : grub2-tools
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch       : 1
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Version     : 2.02
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Release     : 0.76.el7.centos
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Tue 05 Nov 2019 03:40:24 PM UTC
2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Group       : System Environment/Base
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Size        : 10080361
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:License     : GPLv3+
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Signature   : RSA/SHA256, Mon 12 Nov 2018 02:28:11 PM UTC, Key ID 24c6a8a7f4a80eb5
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM  : grub2-2.02-0.76.el7.centos.src.rpm
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date  : Fri 09 Nov 2018 12:58:43 AM UTC
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host  : kbuilder.bsys.centos.org
2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable)
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Packager    : CentOS BuildSystem <http://bugs.centos.org>
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor      : CentOS
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:URL         : http://www.gnu.org/software/grub/
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Summary     : Support tools for GRUB.
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Description :
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:
2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and
2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture.  It supports a rich
2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and
2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices.
2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:
2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms.
2019-12-15T03:17:24.568 INFO:teuthology.task.kernel:Updating Grub Version: grub2
2019-12-15T03:17:24.568 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:24.568 INFO:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg
2019-12-15T03:17:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ...
2019-12-15T03:17:25.173 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c
2019-12-15T03:17:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-5.5.0-rc1-ceph-g7943cf18163c.img
2019-12-15T03:17:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64
2019-12-15T03:17:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img
2019-12-15T03:17:25.531 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64
2019-12-15T03:17:25.552 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img
2019-12-15T03:17:25.585 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003
2019-12-15T03:17:25.596 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img
2019-12-15T03:17:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:done
2019-12-15T03:17:27.021 INFO:teuthology.orchestra.run.smithi067:> mktemp
2019-12-15T03:17:27.090 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.z7Yw7c6m5H
2019-12-15T03:17:27.090 INFO:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.z7Yw7c6m5H
2019-12-15T03:17:27.175 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.z7Yw7c6m5H
2019-12-15T03:17:27.330 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.z7Yw7c6m5H is 6KB
2019-12-15T03:17:27.391 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.z7Yw7c6m5H
2019-12-15T03:17:27.441 INFO:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true
2019-12-15T03:17:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/ls: cannot access /boot/loader/entries: No such file or directory
2019-12-15T03:17:27.591 WARNING:teuthology.task.kernel:Unable to update grub2 order
...
2019-12-15T03:20:25.096 ERROR:teuthology.task.kernel:Saw exception
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 697, in wait_for_reboot
    'failed to install new kernel version within timeout'
AssertionError: failed to install new kernel version within timeout

See: /ceph/teuthology-archive/pdonnell-2019-12-15_02:15:20-kcephfs-wip-pdonnell-testing-20191212.003246-distro-basic-smithi/4603742/teuthology.log

Also:

Failure: failed to install new kernel version within timeout
18 jobs: ['4603742', '4603761', '4603724', '4603757', '4603748', '4603718', '4603730', '4603723', '4603727', '4603740', '4603709', '4603711', '4603745', '4603714', '4603739', '4603743', '4603746', '4603749']
suites intersection: ['conf/{client.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{distro/random/{k-testing.yaml', 'overrides/{frag_enable.yaml', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
suites union: ['clusters/1-mds-1-client.yaml', 'clusters/1-mds-4-client.yaml', 'conf/{client.yaml', 'inline/no.yaml', 'kcephfs/cephfs/{begin.yaml', 'kcephfs/recovery/{begin.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'objectstore-ec/bluestore-bitmap.yaml', 'objectstore-ec/bluestore-comp-ec-root.yaml', 'objectstore-ec/bluestore-comp.yaml', 'objectstore-ec/bluestore-ec-root.yaml', 'objectstore-ec/filestore-xfs.yaml', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{distro/random/{k-testing.yaml', 'overrides/{frag_enable.yaml', 'supported$/{centos_7.yaml}}', 'supported$/{rhel_7.yaml}}', 'tasks/client-limits.yaml}', 'tasks/client-recovery.yaml}', 'tasks/damage.yaml}', 'tasks/data-scan.yaml}', 'tasks/failover.yaml}', 'tasks/journal-repair.yaml}', 'tasks/kclient_workunit_suites_ffsb.yaml}', 'tasks/mds-flush.yaml}', 'tasks/mds-full.yaml}', 'tasks/sessionmap.yaml}', 'tasks/strays.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']

History

#1 Updated by Sage Weil about 3 years ago

  • Status changed from New to Fix Under Review

#2 Updated by Sage Weil about 3 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF