Bug #14995
closed"ceph version 10.0.3 was not installed, found 10.0.4."
Added by Greg Farnum about 8 years ago. Updated almost 8 years ago.
0%
Description
We're seeing this in a lot of runs. It might just be a consequence of scheduling across release tags, but I'm not sure. For instance, http://pulpito.ceph.com/teuthology-2016-03-05_18:04:02-fs-master---basic-smithi/42528/
2016-03-05T19:17:09.492 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/6018ccd6c4405c6014c65dd92660898adbc29c03/version
2016-03-05T19:17:09.498 INFO:teuthology.packaging:Package found...
2016-03-05T19:17:09.498 INFO:teuthology.packaging:Found version: 10.0.3
2016-03-05T19:17:09.499 INFO:teuthology.orchestra.run.smithi001:Running: "rpm -q ceph --qf '%{VERSION}'"
2016-03-05T19:17:09.601 INFO:teuthology.packaging:The installed version of ceph is 10.0.4
2016-03-05T19:17:09.601 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 645, in install
install_packages(ctx, install_info, config)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 390, in install_packages
verify_package_version(ctx, config, remote)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 326, in verify_package_version
pkg=pkg_to_check
RuntimeError: ceph version 10.0.3 was not installed, found 10.0.4.
Updated by Zack Cerza about 8 years ago
- Project changed from teuthology to devops
This is either a build or packaging issue.
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/6018ccd6c4405c6014c65dd92660898adbc29c03/version says 'v10.0.3-2722.g6018ccd'
But:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/6018ccd6c4405c6014c65dd92660898adbc29c03/x86_64/
has packages like:
http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/6018ccd6c4405c6014c65dd92660898adbc29c03/x86_64/ceph-10.0.4-2722.g6018ccd.el7.x86_64.rpm
Updated by Sage Weil about 8 years ago
the tag wasn't pushed, but is now. the branch being tested just needs to be rebuilt.
Updated by Yuri Weinstein about 8 years ago
- Status changed from Resolved to 12
Updated by Dan Mick about 8 years ago
Investigated; from IRC:
http://pulpito.ceph.com/teuthology-2016-03-09_02:10:10-upgrade:infernalis-x-jewel-distro-basic-vps/48622 failed because the sha1 it got for jewel, 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d, for ceph-rpm-centos7, was built/rsynced on 3/4 (see http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d/). The version there is 10.0.3, meaning it was built before the annotated tag was pushed on Monday 3/7. Between then and the time of the run, jewel wasn't rebuilt on centos7, because nothing had been pushed.
Now, there's a further update to jewel, so it will be rebuilding everywhere, and version should update from the now-present annotated tag.
Moral: short-circuiting the build process manually is a really horrible idea, the equivalent of "we don't have time to stop for gas".
Updated by Yuri Weinstein almost 8 years ago
- Status changed from Resolved to New
Reopened, same issue for 10.2.1 across several runs on jewel.
One example run http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-05-15_02:10:02-upgrade:hammer-x-jewel-distro-basic-openstack/
2016-05-15T10:49:18.220 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-15T10:49:18.221 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/3a66dd4f30852819c1bdaa8ec23c795d4ad77269/version
2016-05-15T10:49:18.408 INFO:teuthology.packaging:Package found...
2016-05-15T10:49:18.410 INFO:teuthology.packaging:Found version: 10.2.0
2016-05-15T10:49:18.411 INFO:teuthology.orchestra.run.target070171:Running: "rpm -q ceph --qf '%{VERSION}'"
2016-05-15T10:49:19.065 INFO:teuthology.packaging:The installed version of ceph is 10.2.1
2016-05-15T10:49:19.066 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
upgrade_common(ctx, config, upgrade_old_style)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
verify_package_version(ctx, node, remote)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.
2016-05-15T10:49:19.091 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=54c67d606fb046fd92a8254add84a5c8
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
upgrade_common(ctx, config, upgrade_old_style)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
verify_package_version(ctx, node, remote)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.
Updated by Yuri Weinstein almost 8 years ago
Run: http://pulpito.ceph.com/yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps/
Jobs: all centos jobs
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps/207721/teuthology.log
2016-05-22T10:35:05.344 INFO:teuthology.orchestra.run.vpm096.stdout:Determining fastest mirrors
2016-05-22T10:35:05.549 INFO:teuthology.orchestra.run.vpm096.stdout: * base: mirror.cogentco.com
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * epel: ftp.linux.ncsu.edu
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * extras: mirrors.centos.webair.com
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * updates: mirror.symnds.com
2016-05-22T10:35:10.164 INFO:teuthology.orchestra.run.vpm096.stdout:24 packages excluded due to repository priority protections
2016-05-22T10:35:10.739 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:libcephfs_jni1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.138 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:rbd-fuse-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.469 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-radosgw-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.749 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:librbd1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.022 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-fuse-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.375 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:python-ceph-compat-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.688 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.043 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-devel-compat-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.336 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-test-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.701 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:librados2-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.990 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:cephfs-java-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:14.366 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:libcephfs1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:14.377 INFO:teuthology.orchestra.run.vpm096.stdout:Nothing to do
2016-05-22T10:35:14.463 INFO:teuthology.orchestra.run.vpm096:Running: 'sudo lsb_release -is'
2016-05-22T10:35:14.531 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-22T10:35:14.532 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489/version
2016-05-22T10:35:14.539 INFO:teuthology.packaging:Package found...
2016-05-22T10:35:14.539 INFO:teuthology.packaging:Found version: 10.2.0
2016-05-22T10:35:14.539 INFO:teuthology.orchestra.run.vpm096:Running: "rpm -q ceph --qf '%{VERSION}'"
2016-05-22T10:35:14.619 INFO:teuthology.packaging:The installed version of ceph is 10.2.1
Updated by Yuri Weinstein almost 8 years ago
- Project changed from devops to Ceph
Updated by Yuri Weinstein almost 8 years ago
- Related to Bug #15927: Upgrade jobs failed "unmet dependencies" error in upgrade:jewel-x-master-distro-basic-vps added
Updated by Yuri Weinstein almost 8 years ago
Updated by Nathan Cutler almost 8 years ago
I studied the log of the job posted by Yuri just now, and noticed the following.
The suite is "upgrade:jewel-x", i.e. first a jewel package will be installed and then it will upgraded to "some x". The full name of the test is yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps which I suppose means that the upgrade target ("some x") is master.
The ceph-qa-suite branch is "master".
Now, the question is which version of jewel will be installed as the base and which version of master will be upgraded to? This is not clear from the YAML.
On line 111 an override to the ceph task is specified:
sha1: 4764b699d6b7f699b9596269c269a99a84f8c489
The same override is specified for the "ceph-deploy", "install", and "workunit" tasks.
And 4764b699d6b7f699b9596269c269a99a84f8c489 is indeed 10.2.0:
$ git describe 4764b699d6b7f699b9596269c269a99a84f8c489 v10.2.0-1219-g4764b69
The "1219" in the git describe output indicates it's 1219 commits after the v10.2.0 tag. I find that it's 200 commits before the v10.2.1 tag:
$ git log --oneline 4764b699d6b7f699b9596269c269a99a84f8c489..v10.2.1 | wc -l 200
The "install" YAML facet says that the base version (pre-upgrade) should be from the jewel branch:
tasks:
- install:
branch: jewel
But of course we have the override which says the SHA1 to install is 4764b699d6b7f699b9596269c269a99a84f8c489.
The "internal" task finds packages for 4764b699d6b7f699b9596269c269a99a84f8c489:
2016-05-22T09:57:53.600 INFO:teuthology.run:suite_path is set to /var/lib/teuthworker/src/ceph-qa-suite_master; will attempt to use it 2016-05-22T09:57:53.601 INFO:teuthology.run:Found tasks at /var/lib/teuthworker/src/ceph-qa-suite_master/tasks 2016-05-22T09:57:53.604 INFO:teuthology.task.internal:Checking packages... 2016-05-22T09:57:53.607 INFO:teuthology.task.internal:Checking packages for os_type,'centos' flavor 'basic' and ceph hash '4764b699d6b7f699b9596269c269a99a84f8c489' 2016-05-22T09:57:53.607 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489/version 2016-05-22T09:57:53.611 INFO:teuthology.packaging:Package found... 2016-05-22T09:57:53.622 INFO:teuthology.packaging:Found version: 10.2.0 2016-05-22T09:57:53.622 INFO:teuthology.task.internal:Found packages for ceph version 10.2.0
When we get to the "install" task, we see evidence that the override is present:
2016-05-22T10:32:34.460 INFO:teuthology.run_tasks:Running task install...
2016-05-22T10:32:34.951 DEBUG:teuthology.task.install:config {'sha1': '4764b699d6b7f699b9596269c269a99a84f8c489', 'branch': 'jewel'}
However, the override is ignored by the install task. Instead of looking in the repo already found by the "internal" task - i.e. the one corresponding to the SHA1 specified in the override - it grabs a different repo:
2016-05-22T10:32:35.212 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel 2016-05-22T10:32:35.212 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/version 2016-05-22T10:32:35.217 INFO:teuthology.packaging:Package found... 2016-05-22T10:32:35.217 INFO:teuthology.packaging:Found version: 10.2.1
Some while later we enter the "install.upgrade" task, which also has the override. Unlike the "install" task, however, it actually respects the override:
2016-05-22T10:34:59.396 INFO:teuthology.run_tasks:Running task install.upgrade...
2016-05-22T10:34:59.433 INFO:teuthology.task.install:project ceph config {'mon.a': None, 'mon.b': None} overrides {'sha1': '4764b699d6b7f699b9596269c269a99a84f8c489'}
...
2016-05-22T10:34:59.629 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-22T10:34:59.629 INFO:teuthology.task.install:Host vpm096 is: centos 7.2.1511 x86_64
2016-05-22T10:34:59.629 INFO:teuthology.task.install:Repo base URL: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489
An attempt to upgrade from 10.2.1 to 10.2.0 is actually a downgrade. Hence the exception.
After completing this analysis, I'm not sure where the bug is, exactly. Yuri, could you show us the teuthology-suite command used to schedule this job? I'm interested to find out where the SHA1 used in the overrides is coming from.
It looks like the "install" task is ignoring the override and installing the latest jewel it can find, while the "install.upgrade" task respects the override. Shouldn't it be the other way around?
One could also say that the test is broken because it is upgrading to SHA1 4764b699d6b7f699b9596269c269a99a84f8c489 (a stale jewel SHA1) when in fact it should be upgrading to the latest master.
What I don't understand is why we need this "sha1: 4764b699d6b7f699b9596269c269a99a84f8c489" override in the first place. It seems like the test would work fine without it - it would install the latest jewel and upgrade to master.
Updated by Yuri Weinstein almost 8 years ago
command used to schedule
teuthology-suite -v -c master -k distro -m vps -s upgrade/jewel-x ~/vps.yaml -p 100
Updated by Yuri Weinstein almost 8 years ago
@Nathan Weinberg, can you take a look again?
See in run: http://pulpito.ceph.com/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/
Jobs - all centos
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/256899/teuthology.log
2016-06-13T12:06:02.311 INFO:teuthology.orchestra.run.vpm115.stdout:Cleaning up list of fastest mirrors
2016-06-13T12:06:02.325 INFO:teuthology.orchestra.run.vpm115:Running: 'sudo yum -y install libcephfs_jni1 rbd-fuse ceph-radosgw librbd1 ceph-fuse python-ceph ceph ceph-devel ceph-test librados2 cephfs-java libcephfs1'
2016-06-13T12:06:02.513 INFO:teuthology.orchestra.run.vpm115.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2016-06-13T12:06:04.872 INFO:teuthology.orchestra.run.vpm115.stdout:Determining fastest mirrors
2016-06-13T12:06:08.032 INFO:teuthology.orchestra.run.vpm115.stdout: * base: mirror.umd.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * epel: ftp.linux.ncsu.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * extras: mirror.lug.udel.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * updates: mirror.cc.columbia.edu
2016-06-13T12:06:12.791 INFO:teuthology.orchestra.run.vpm115.stdout:24 packages excluded due to repository priority protections
2016-06-13T12:06:13.377 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:libcephfs_jni1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:13.792 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:rbd-fuse-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.128 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-radosgw-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.436 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:librbd1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.758 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-fuse-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:15.137 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:python-ceph-compat-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
149952016-06-13T12:06:15.439 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:15.813 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-devel-compat-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.092 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-test-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.474 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:librados2-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.768 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:cephfs-java-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking 14995for update.
2016-06-13T12:06:17.14995135 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:libcephfs1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:17.135 INFO:teuthology.orchestra.run.vpm115.stdout:Nothing to do
2016-06-13T12:06:17.229 INFO:teuthology.orchestra.run.vpm115:Running: 'sudo lsb_release -is'
2016-06-13T12:06:17.298 DEBUG:teuthology.misc:System to be installed: CentOS
2016-06-13T12:06:17.298 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/082f918a99807af297f75b7eb0fda627e24725f1/version
2016-06-13T12:06:17.303 INFO:teuthology.packaging:Package found...
2016-06-13T12:06:17.303 INFO:teuthology.packaging:Found version: 10.2.0
2016-06-13T12:06:17.304 INFO:teuthology.orchestra.run.vpm115:Running: "rpm -q ceph --qf '%{VERSION}'"
2016-06-13T12:06:17.384 INFO:teuthology.packaging:The installed version of ceph is 10.2.1
2016-06-13T12:06:17.384 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
upgrade_common(ctx, config, upgrade_old_style)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
149951499514995 verify_package_version(ctx, node, remote)
File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.
and on ubuntu:
Job: 256903
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/256903/teuthology.log
2016-06-13T11:50:10.281 INFO:teuthology.orchestra.run.vpm011.stdout:Reading package lists... 2016-06-13T11:50:10.285 INFO:teuthology.orchestra.run.vpm011:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=10.2.0-2260-g082f918-1trusty rbd-fuse=10.2.0-2260-g082f918-1trusty librbd1=10.2.0-2260-g082f918-1trusty ceph-fuse=10.2.0-2260-g082f918-1trusty python-ceph=10.2.0-2260-g082f918-1trusty ceph-common=10.2.0-2260-g082f918-1trusty libcephfs-java=10.2.0-2260-g082f918-1trusty ceph=10.2.0-2260-g082f918-1trusty libcephfs-jni=10.2.0-2260-g082f918-1trusty ceph-test=10.2.0-2260-g082f918-1trusty radosgw=10.2.0-2260-g082f918-1trusty librados2=10.2.0-2260-g082f918-1trusty libcephfs1=10.2.0-2260-g082f918-1trusty' 2016-06-13T11:50:10.307 INFO:teuthology.orchestra.run.vpm011.stdout:Reading package lists... 2016-06-13T11:50:10.469 INFO:teuthology.orchestra.run.vpm011.stdout:Building dependency tree... 2016-06-13T11:50:10.470 INFO:teuthology.orchestra.run.vpm011.stdout:Reading state information... 2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:Some packages could not be installed. This may mean that you have 2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:requested an impossible situation or if you are using the unstable 2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:distribution that some required packages have not yet been created 2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout:or been moved out of Incoming. 2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout:The following information may help to resolve the situation: 2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout: 2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout:The following packages have unmet dependencies: 2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout: ceph-common : Depends: librbd1 (= 10.2.1-241-g1720664-1trusty) but 10.2.0-2260-g082f918-1trusty is to be installed 2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: libradosstriper1 but it is not going to be installed 2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: librgw2 but it is not going to be installed 2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: python-rados (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2149951499514995016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: python-cephfs (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: python-rbd (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout: ceph-test : Depends: libradosstriper1 but it is not going to be installed 2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: librgw2 but it is not going to be installed 2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout: python-ceph : Depends: python-rados (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: python-rbd (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.590 INFO:teuthology.orchestra.run.vpm011.stdout: Depends: python-cephfs (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.590 INFO:teuthology.orchestra.run.vpm011.stdout: radosgw : Depends: librgw2 (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed 2016-06-13T11:50:10.601 INFO:teuthology.orchestra.run.vpm011.stderr:E: Unable to correct problems, you have held broken packages.
Updated by Nathan Cutler almost 8 years ago
- Status changed from New to 12
- Assignee set to Nathan Cutler
Updated by Nathan Cutler almost 8 years ago
I think these problems are caused by tags being present in one branch, but absent in another, combined with the use of "git describe" to determine the package (rpm, deb) version.
Consider the following example:
First, I install jewel 10.2.1. The package version string starts with "10.2.1". This is derived from the output of "git describe". For example, in comment #14995-14 above we have ceph packages with version string "10.2.1-241-g1720664-1trusty". That means the packages were built from SHA1 1720664 which is 241 commits ahead of tag v10.2.1. Please note that the "v10.2.1" tag exists only in the jewel branch.
Next, I attempt to upgrade to master, which has a "v10.2.0" tag but no "v10.2.1" tag and indeed "v10.2.0" is the latest tag in master. As a result, "git describe" on current master returns:
v10.2.0-2846-g5de4a9c
And of course gitbuilder packages built from the master branch have a version that starts with "10.2.0" and the package managers will refuse to upgrade from "10.2.1" to "10.2.0".
The jewel -> master upgrade scenario will work as long as "v10.2.0" is the highest tag in both branches. Once jewel gets a "v10.2.1" tag, an upgrade from jewel to master will be impossible until master gets a v11.0.0 tag.
I suspect all instances of this bug are caused by variations on this theme.
As a fix, a sanity check could be added at the beginning of the upgrade suite. It would assert that the destination version string is greater than the source version string. If this fails, the suite would fail with a clear error right at the beginning.
Updated by Nathan Cutler almost 8 years ago
- Status changed from 12 to Resolved
Sage added the v11.0.0 tag to master and this tag will show up in kraken, too, when master is merged into kraken.
I opened #16521 for adding the pre-upgrade sanity check.