Actions
Bug #11957
closed'AssertionError: failed to install new kernel version within timeout'
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/firefly
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2015-06-10_15:12:55-upgrade:firefly-firefly---basic-vps/
Jobs: run is still in progress, suspect all will fail
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-06-10_15:12:55-upgrade:firefly-firefly---basic-vps/928400/
2015-06-10T15:51:00.132 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version...
2015-06-10T15:51:00.132 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want v3.18-rc4-15-g388cfdc...
2015-06-10T15:51:00.133 INFO:teuthology.orchestra.run.vpm099:Running: 'uname -r'
2015-06-10T15:51:00.171 DEBUG:teuthology.task.kernel:current kernel version is 3.18.0-rc4-ceph-00015-g388cfdc
2015-06-10T15:51:00.172 ERROR:teuthology.task.kernel:Saw exception
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 605, in wait_for_reboot
'failed to install new kernel version within timeout'
AssertionError: failed to install new kernel version within timeout
2015-06-10T15:51:00.173 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version...
2015-06-10T15:51:00.173 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want v3.18-rc4-15-g388cfdc...
2015-06-10T15:51:00.173 INFO:teuthology.orchestra.run.vpm102:Running: 'uname -r'
2015-06-10T15:51:00.196 DEBUG:teuthology.task.kernel:current kernel version is 3.18.0-rc4-ceph-00015-g388cfdc
2015-06-10T15:51:00.196 ERROR:teuthology.task.kernel:Saw exception
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 605, in wait_for_reboot
'failed to install new kernel version within timeout'
AssertionError: failed to install new kernel version within timeout
2015-06-10T15:51:00.196 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version...
2015-06-10T15:51:00.197 INFO:teuthology.task.kernel:Checking kernel version of client.0, want v3.18-rc4-15-g388cfdc...
2015-06-10T15:51:00.197 ERROR:teuthology.task.kernel:Saw exception
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 604, in wait_for_reboot
assert not need_to_install(ctx, client, need_install[client]), \
File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 148, in need_to_install
stdout=uname_fp,
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 333, in run
raise ConnectionLostError(command=quote(args), node=name)
ConnectionLostError: SSH connection to vpm047 was lost: 'uname -r'
Updated by Yuri Weinstein almost 9 years ago
Updated by Yuri Weinstein almost 9 years ago
Updated by Yuri Weinstein almost 9 years ago
- Status changed from Resolved to New
Updated by Zack Cerza almost 9 years ago
- Status changed from New to 12
- Assignee set to Ilya Dryomov
I see a couple issues here. One is:
2015-06-16T10:20:11.043 INFO:teuthology.task.kernel:Installing kernel 388cfdc9bc19cc189b86770dba55910018e8e0fc on mon.a... 2015-06-16T10:20:11.043 INFO:teuthology.orchestra.run.vpm146:Running: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm' 2015-06-16T10:20:11.107 INFO:teuthology.orchestra.run.vpm146.stderr:warning: /tmp/kernel.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 03c3951a: NOKEY 2015-06-16T10:20:11.475 INFO:teuthology.orchestra.run.vpm146.stdout:Preparing... ################################################## 2015-06-16T10:20:46.128 INFO:teuthology.orchestra.run.vpm146.stdout:kernel ################################################## 2015-06-16T10:20:57.296 INFO:teuthology.orchestra.run.vpm146.stderr:/sbin/new-kernel-pkg: line 310: 2283 Killed depmod -ae -F /boot/System.map-$version $version 2015-06-16T10:20:59.260 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module tun 2015-06-16T10:20:59.280 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module ipv6 2015-06-16T10:20:59.302 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module snd_page_alloc 2015-06-16T10:20:59.303 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module virtio_net 2015-06-16T10:20:59.304 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module i2c_core 2015-06-16T10:20:59.305 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module sg 2015-06-16T10:20:59.305 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module ext4 2015-06-16T10:20:59.306 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module jbd2 2015-06-16T10:20:59.306 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module mbcache 2015-06-16T10:20:59.307 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module virtio_blk 2015-06-16T10:20:59.307 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module sr_mod 2015-06-16T10:20:59.308 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module cdrom 2015-06-16T10:20:59.308 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module virtio_pci 2015-06-16T10:20:59.309 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module virtio_ring 2015-06-16T10:20:59.309 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module virtio 2015-06-16T10:20:59.311 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module ata_generic 2015-06-16T10:20:59.311 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module ata_piix 2015-06-16T10:20:59.315 INFO:teuthology.orchestra.run.vpm146.stderr:ERROR: modinfo: could not find module dm_mod 2015-06-16T10:21:03.217 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@vpm146.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None)
The other is:
2015-06-16T10:24:21.054 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2015-06-16T10:24:21.054 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want v3.18-rc4-15-g388cfdc... 2015-06-16T10:24:21.055 INFO:teuthology.orchestra.run.vpm146:Running: 'uname -r' 2015-06-16T10:24:21.078 DEBUG:teuthology.task.kernel:current kernel version is 3.18.0-rc4-ceph-00015-g388cfdc
The above versions are very similar, but not identical. We need them to match up.
Ilya, would you take a look at this please?
Updated by Ilya Dryomov almost 9 years ago
- Status changed from 12 to In Progress
Not sure about the real cause of either of the issues, but this is centos6 and kernel centos6 gitbuilder has been dead for a while - its master branch points to 3.18-something when 4.1 is around the corner. I checked the centos7 gitbuilder, which is the one we care about, and there the version strings are correct.
Now to why this happened: it's still the kernel task flavor thing. I looked into kernel, install and upgrade task configs and came to the conclusion that the syntax needs to be unified before they are switched to the new Task class. I abandoned my kernel-only patches and started going through all the details there. That turned out to be pretty time consuming as I don't know that much about install and upgrade tasks and I got distracted with krbd work.
Yuri, the summary is that this is a kernel task / teuthology-suite problem - this job config shouldn't have invoked kernel task. The fact that it got invoked is a bug. I'll do a PR with short term fix tomorrow.
Updated by Yuri Weinstein almost 9 years ago
Thx Ilya. Let me know when it's ready and I will retest. This is a part of firefly v0.80.10 validation, don't think it should block anything, but would like to have it resolved this before it is published.
Updated by Ilya Dryomov almost 9 years ago
- Status changed from In Progress to Fix Under Review
Updated by Ilya Dryomov almost 9 years ago
- Status changed from Fix Under Review to Resolved
Updated by Yuri Weinstein almost 9 years ago
- Status changed from Resolved to 12
Updated by Yuri Weinstein almost 9 years ago
- Status changed from 12 to Resolved
Updated by Laura Flores 12 months ago
- Related to Bug #61159: AssertionError: failed to install new kernel version within timeout added
Actions