Bug #43335
AssertionError: failed to install new kernel version within timeout
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2019-12-15T03:17:24.232 DEBUG:teuthology.task.kernel:get_image_version: 5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:24.232 INFO:teuthology.orchestra.run.smithi067:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2019-12-15T03:17:24.374 INFO:teuthology.orchestra.run.smithi067.stdout:postinstall scriptlet (using /bin/sh): 2019-12-15T03:17:24.374 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c -a -r /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c ]; then 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:cp /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:cp /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:rm -f /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c /boot/System.map-5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/installkernel 5.5.0-rc1-ceph-g7943cf18163c /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:rm -f /boot/.vmlinuz-5.5.0-rc1-ceph-g7943cf18163c-rpm /boot/.System.map-5.5.0-rc1-ceph-g7943cf18163c-rpm 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2019-12-15T03:17:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:preuninstall scriptlet (using /bin/sh): 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:new-kernel-pkg --remove 5.5.0-rc1-ceph-g7943cf18163c --rminitrd --initrdfile=/boot/initramfs-5.5.0-rc1-ceph-g7943cf18163c.img 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:elif [ -x /usr/bin/kernel-install ]; then 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-install remove 5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:postuninstall scriptlet (using /bin/sh): 2019-12-15T03:17:24.376 INFO:teuthology.orchestra.run.smithi067.stdout:if [ -x /sbin/update-bootloader ]; then 2019-12-15T03:17:24.377 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/update-bootloader --remove 5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:24.377 INFO:teuthology.orchestra.run.smithi067.stdout:fi 2019-12-15T03:17:24.378 INFO:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2019-12-15T03:17:24.563 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.02 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 0.76.el7.centos 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Tue 05 Nov 2019 03:40:24 PM UTC 2019-12-15T03:17:24.564 INFO:teuthology.orchestra.run.smithi067.stdout:Group : System Environment/Base 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 10080361 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:11 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-15T03:17:24.565 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable) 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : CentOS BuildSystem <http://bugs.centos.org> 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-15T03:17:24.566 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-15T03:17:24.567 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2019-12-15T03:17:24.568 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-15T03:17:24.568 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:24.568 INFO:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-15T03:17:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2019-12-15T03:17:25.173 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.5.0-rc1-ceph-g7943cf18163c 2019-12-15T03:17:25.187 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-5.5.0-rc1-ceph-g7943cf18163c.img 2019-12-15T03:17:25.458 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-15T03:17:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-15T03:17:25.531 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-15T03:17:25.552 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-15T03:17:25.585 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-15T03:17:25.596 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-15T03:17:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:done 2019-12-15T03:17:27.021 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-15T03:17:27.090 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.z7Yw7c6m5H 2019-12-15T03:17:27.090 INFO:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.z7Yw7c6m5H 2019-12-15T03:17:27.175 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.z7Yw7c6m5H 2019-12-15T03:17:27.330 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.z7Yw7c6m5H is 6KB 2019-12-15T03:17:27.391 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.z7Yw7c6m5H 2019-12-15T03:17:27.441 INFO:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2019-12-15T03:17:27.589 INFO:teuthology.orchestra.run.smithi067.stderr:/bin/ls: cannot access /boot/loader/entries: No such file or directory 2019-12-15T03:17:27.591 WARNING:teuthology.task.kernel:Unable to update grub2 order ... 2019-12-15T03:20:25.096 ERROR:teuthology.task.kernel:Saw exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 697, in wait_for_reboot 'failed to install new kernel version within timeout' AssertionError: failed to install new kernel version within timeout
See: /ceph/teuthology-archive/pdonnell-2019-12-15_02:15:20-kcephfs-wip-pdonnell-testing-20191212.003246-distro-basic-smithi/4603742/teuthology.log
Also:
Failure: failed to install new kernel version within timeout 18 jobs: ['4603742', '4603761', '4603724', '4603757', '4603748', '4603718', '4603730', '4603723', '4603727', '4603740', '4603709', '4603711', '4603745', '4603714', '4603739', '4603743', '4603746', '4603749'] suites intersection: ['conf/{client.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{distro/random/{k-testing.yaml', 'overrides/{frag_enable.yaml', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}'] suites union: ['clusters/1-mds-1-client.yaml', 'clusters/1-mds-4-client.yaml', 'conf/{client.yaml', 'inline/no.yaml', 'kcephfs/cephfs/{begin.yaml', 'kcephfs/recovery/{begin.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'objectstore-ec/bluestore-bitmap.yaml', 'objectstore-ec/bluestore-comp-ec-root.yaml', 'objectstore-ec/bluestore-comp.yaml', 'objectstore-ec/bluestore-ec-root.yaml', 'objectstore-ec/filestore-xfs.yaml', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{distro/random/{k-testing.yaml', 'overrides/{frag_enable.yaml', 'supported$/{centos_7.yaml}}', 'supported$/{rhel_7.yaml}}', 'tasks/client-limits.yaml}', 'tasks/client-recovery.yaml}', 'tasks/damage.yaml}', 'tasks/data-scan.yaml}', 'tasks/failover.yaml}', 'tasks/journal-repair.yaml}', 'tasks/kclient_workunit_suites_ffsb.yaml}', 'tasks/mds-flush.yaml}', 'tasks/mds-full.yaml}', 'tasks/sessionmap.yaml}', 'tasks/strays.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
Related issues
History
#1 Updated by Sage Weil over 4 years ago
- Status changed from New to Fix Under Review
#2 Updated by Sage Weil over 4 years ago
- Status changed from Fix Under Review to Resolved
#3 Updated by Laura Flores 11 months ago
- Related to Bug #61159: AssertionError: failed to install new kernel version within timeout added