Project

General

Profile

Bug #14995

"ceph version 10.0.3 was not installed, found 10.0.4."

Added by Greg Farnum over 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
03/07/2016
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

We're seeing this in a lot of runs. It might just be a consequence of scheduling across release tags, but I'm not sure. For instance, http://pulpito.ceph.com/teuthology-2016-03-05_18:04:02-fs-master---basic-smithi/42528/

2016-03-05T19:17:09.492 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/6018ccd6c4405c6014c65dd92660898adbc29c03/version
2016-03-05T19:17:09.498 INFO:teuthology.packaging:Package found...
2016-03-05T19:17:09.498 INFO:teuthology.packaging:Found version: 10.0.3
2016-03-05T19:17:09.499 INFO:teuthology.orchestra.run.smithi001:Running: "rpm -q ceph --qf '%{VERSION}'" 
2016-03-05T19:17:09.601 INFO:teuthology.packaging:The installed version of ceph is 10.0.4
2016-03-05T19:17:09.601 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 645, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 390, in install_packages
    verify_package_version(ctx, config, remote)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 326, in verify_package_version
    pkg=pkg_to_check
RuntimeError: ceph version 10.0.3 was not installed, found 10.0.4.

Related issues

Related to Ceph - Bug #15927: Upgrade jobs failed "unmet dependencies" error in upgrade:jewel-x-master-distro-basic-vps Can't reproduce 05/18/2016

History

#2 Updated by Sage Weil over 3 years ago

the tag wasn't pushed, but is now. the branch being tested just needs to be rebuilt.

#3 Updated by Sage Weil over 3 years ago

  • Status changed from New to Resolved

#5 Updated by Dan Mick over 3 years ago

Investigated; from IRC:

http://pulpito.ceph.com/teuthology-2016-03-09_02:10:10-upgrade:infernalis-x-jewel-distro-basic-vps/48622 failed because the sha1 it got for jewel, 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d, for ceph-rpm-centos7, was built/rsynced on 3/4 (see http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d/). The version there is 10.0.3, meaning it was built before the annotated tag was pushed on Monday 3/7. Between then and the time of the run, jewel wasn't rebuilt on centos7, because nothing had been pushed.

Now, there's a further update to jewel, so it will be rebuilding everywhere, and version should update from the now-present annotated tag.

Moral: short-circuiting the build process manually is a really horrible idea, the equivalent of "we don't have time to stop for gas".

#6 Updated by Dan Mick over 3 years ago

  • Status changed from Verified to Resolved

#7 Updated by Yuri Weinstein over 3 years ago

  • Status changed from Resolved to New

Reopened, same issue for 10.2.1 across several runs on jewel.
One example run http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-05-15_02:10:02-upgrade:hammer-x-jewel-distro-basic-openstack/

2016-05-15T10:49:18.220 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-15T10:49:18.221 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/3a66dd4f30852819c1bdaa8ec23c795d4ad77269/version
2016-05-15T10:49:18.408 INFO:teuthology.packaging:Package found...
2016-05-15T10:49:18.410 INFO:teuthology.packaging:Found version: 10.2.0
2016-05-15T10:49:18.411 INFO:teuthology.orchestra.run.target070171:Running: "rpm -q ceph --qf '%{VERSION}'" 
2016-05-15T10:49:19.065 INFO:teuthology.packaging:The installed version of ceph is 10.2.1
2016-05-15T10:49:19.066 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
    upgrade_common(ctx, config, upgrade_old_style)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
    verify_package_version(ctx, node, remote)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
    pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.
2016-05-15T10:49:19.091 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=54c67d606fb046fd92a8254add84a5c8
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
    upgrade_common(ctx, config, upgrade_old_style)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
    verify_package_version(ctx, node, remote)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
    pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.

#8 Updated by Yuri Weinstein over 3 years ago

Run: http://pulpito.ceph.com/yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps/
Jobs: all centos jobs
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps/207721/teuthology.log

2016-05-22T10:35:05.344 INFO:teuthology.orchestra.run.vpm096.stdout:Determining fastest mirrors
2016-05-22T10:35:05.549 INFO:teuthology.orchestra.run.vpm096.stdout: * base: mirror.cogentco.com
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * epel: ftp.linux.ncsu.edu
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * extras: mirrors.centos.webair.com
2016-05-22T10:35:05.550 INFO:teuthology.orchestra.run.vpm096.stdout: * updates: mirror.symnds.com
2016-05-22T10:35:10.164 INFO:teuthology.orchestra.run.vpm096.stdout:24 packages excluded due to repository priority protections
2016-05-22T10:35:10.739 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:libcephfs_jni1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.138 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:rbd-fuse-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.469 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-radosgw-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:11.749 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:librbd1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.022 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-fuse-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.375 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:python-ceph-compat-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:12.688 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.043 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-devel-compat-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.336 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:ceph-test-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.701 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:librados2-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:13.990 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:cephfs-java-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:14.366 INFO:teuthology.orchestra.run.vpm096.stdout:Package matching 1:libcephfs1-10.2.0-1219.g4764b69.el7.x86_64 already installed. Checking for update.
2016-05-22T10:35:14.377 INFO:teuthology.orchestra.run.vpm096.stdout:Nothing to do
2016-05-22T10:35:14.463 INFO:teuthology.orchestra.run.vpm096:Running: 'sudo lsb_release -is'
2016-05-22T10:35:14.531 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-22T10:35:14.532 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489/version
2016-05-22T10:35:14.539 INFO:teuthology.packaging:Package found...
2016-05-22T10:35:14.539 INFO:teuthology.packaging:Found version: 10.2.0
2016-05-22T10:35:14.539 INFO:teuthology.orchestra.run.vpm096:Running: "rpm -q ceph --qf '%{VERSION}'" 
2016-05-22T10:35:14.619 INFO:teuthology.packaging:The installed version of ceph is 10.2.1

#9 Updated by Yuri Weinstein over 3 years ago

  • Project changed from devops to Ceph

#10 Updated by Yuri Weinstein over 3 years ago

  • Related to Bug #15927: Upgrade jobs failed "unmet dependencies" error in upgrade:jewel-x-master-distro-basic-vps added

#11 Updated by Yuri Weinstein over 3 years ago

#14995 and #15927 seem related as one issue is on CentOS and another is on Trusty.

#12 Updated by Nathan Cutler over 3 years ago

I studied the log of the job posted by Yuri just now, and noticed the following.

The suite is "upgrade:jewel-x", i.e. first a jewel package will be installed and then it will upgraded to "some x". The full name of the test is yuriw-2016-05-22_09:55:50-upgrade:jewel-x-master-distro-basic-vps which I suppose means that the upgrade target ("some x") is master.

The ceph-qa-suite branch is "master".

Now, the question is which version of jewel will be installed as the base and which version of master will be upgraded to? This is not clear from the YAML.

On line 111 an override to the ceph task is specified:

sha1: 4764b699d6b7f699b9596269c269a99a84f8c489

The same override is specified for the "ceph-deploy", "install", and "workunit" tasks.

And 4764b699d6b7f699b9596269c269a99a84f8c489 is indeed 10.2.0:

$ git describe 4764b699d6b7f699b9596269c269a99a84f8c489
v10.2.0-1219-g4764b69

The "1219" in the git describe output indicates it's 1219 commits after the v10.2.0 tag. I find that it's 200 commits before the v10.2.1 tag:

$ git log --oneline 4764b699d6b7f699b9596269c269a99a84f8c489..v10.2.1 | wc -l
200

The "install" YAML facet says that the base version (pre-upgrade) should be from the jewel branch:

  tasks:
  - install:
      branch: jewel

But of course we have the override which says the SHA1 to install is 4764b699d6b7f699b9596269c269a99a84f8c489.

The "internal" task finds packages for 4764b699d6b7f699b9596269c269a99a84f8c489:

2016-05-22T09:57:53.600 INFO:teuthology.run:suite_path is set to /var/lib/teuthworker/src/ceph-qa-suite_master; will attempt to use it
2016-05-22T09:57:53.601 INFO:teuthology.run:Found tasks at /var/lib/teuthworker/src/ceph-qa-suite_master/tasks
2016-05-22T09:57:53.604 INFO:teuthology.task.internal:Checking packages...
2016-05-22T09:57:53.607 INFO:teuthology.task.internal:Checking packages for os_type,'centos' flavor 'basic' and ceph hash '4764b699d6b7f699b9596269c269a99a84f8c489'
2016-05-22T09:57:53.607 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489/version
2016-05-22T09:57:53.611 INFO:teuthology.packaging:Package found...
2016-05-22T09:57:53.622 INFO:teuthology.packaging:Found version: 10.2.0
2016-05-22T09:57:53.622 INFO:teuthology.task.internal:Found packages for ceph version 10.2.0

When we get to the "install" task, we see evidence that the override is present:

2016-05-22T10:32:34.460 INFO:teuthology.run_tasks:Running task install...
2016-05-22T10:32:34.951 DEBUG:teuthology.task.install:config {'sha1': '4764b699d6b7f699b9596269c269a99a84f8c489', 'branch': 'jewel'}

However, the override is ignored by the install task. Instead of looking in the repo already found by the "internal" task - i.e. the one corresponding to the SHA1 specified in the override - it grabs a different repo:

2016-05-22T10:32:35.212 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel
2016-05-22T10:32:35.212 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/jewel/version
2016-05-22T10:32:35.217 INFO:teuthology.packaging:Package found...
2016-05-22T10:32:35.217 INFO:teuthology.packaging:Found version: 10.2.1

Some while later we enter the "install.upgrade" task, which also has the override. Unlike the "install" task, however, it actually respects the override:

2016-05-22T10:34:59.396 INFO:teuthology.run_tasks:Running task install.upgrade...
2016-05-22T10:34:59.433 INFO:teuthology.task.install:project ceph config {'mon.a': None, 'mon.b': None} overrides {'sha1': '4764b699d6b7f699b9596269c269a99a84f8c489'}
...
2016-05-22T10:34:59.629 DEBUG:teuthology.misc:System to be installed: CentOS
2016-05-22T10:34:59.629 INFO:teuthology.task.install:Host vpm096 is: centos 7.2.1511 x86_64
2016-05-22T10:34:59.629 INFO:teuthology.task.install:Repo base URL: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/4764b699d6b7f699b9596269c269a99a84f8c489

An attempt to upgrade from 10.2.1 to 10.2.0 is actually a downgrade. Hence the exception.

After completing this analysis, I'm not sure where the bug is, exactly. Yuri, could you show us the teuthology-suite command used to schedule this job? I'm interested to find out where the SHA1 used in the overrides is coming from.

It looks like the "install" task is ignoring the override and installing the latest jewel it can find, while the "install.upgrade" task respects the override. Shouldn't it be the other way around?

One could also say that the test is broken because it is upgrading to SHA1 4764b699d6b7f699b9596269c269a99a84f8c489 (a stale jewel SHA1) when in fact it should be upgrading to the latest master.

What I don't understand is why we need this "sha1: 4764b699d6b7f699b9596269c269a99a84f8c489" override in the first place. It seems like the test would work fine without it - it would install the latest jewel and upgrade to master.

#13 Updated by Yuri Weinstein over 3 years ago

command used to schedule

teuthology-suite -v -c master -k distro -m vps -s upgrade/jewel-x ~/vps.yaml -p 100

#14 Updated by Yuri Weinstein about 3 years ago

@Nathan, can you take a look again?

See in run: http://pulpito.ceph.com/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/
Jobs - all centos
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/256899/teuthology.log

2016-06-13T12:06:02.311 INFO:teuthology.orchestra.run.vpm115.stdout:Cleaning up list of fastest mirrors
2016-06-13T12:06:02.325 INFO:teuthology.orchestra.run.vpm115:Running: 'sudo yum -y install libcephfs_jni1 rbd-fuse ceph-radosgw librbd1 ceph-fuse python-ceph ceph ceph-devel ceph-test librados2 cephfs-java libcephfs1'
2016-06-13T12:06:02.513 INFO:teuthology.orchestra.run.vpm115.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2016-06-13T12:06:04.872 INFO:teuthology.orchestra.run.vpm115.stdout:Determining fastest mirrors
2016-06-13T12:06:08.032 INFO:teuthology.orchestra.run.vpm115.stdout: * base: mirror.umd.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * epel: ftp.linux.ncsu.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * extras: mirror.lug.udel.edu
2016-06-13T12:06:08.033 INFO:teuthology.orchestra.run.vpm115.stdout: * updates: mirror.cc.columbia.edu
2016-06-13T12:06:12.791 INFO:teuthology.orchestra.run.vpm115.stdout:24 packages excluded due to repository priority protections
2016-06-13T12:06:13.377 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:libcephfs_jni1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:13.792 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:rbd-fuse-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.128 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-radosgw-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.436 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:librbd1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:14.758 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-fuse-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:15.137 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:python-ceph-compat-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
149952016-06-13T12:06:15.439 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:15.813 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-devel-compat-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.092 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:ceph-test-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.474 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:librados2-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:16.768 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:cephfs-java-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking 14995for update.
2016-06-13T12:06:17.14995135 INFO:teuthology.orchestra.run.vpm115.stdout:Package matching 1:libcephfs1-10.2.0-2260.g082f918.el7.x86_64 already installed. Checking for update.
2016-06-13T12:06:17.135 INFO:teuthology.orchestra.run.vpm115.stdout:Nothing to do
2016-06-13T12:06:17.229 INFO:teuthology.orchestra.run.vpm115:Running: 'sudo lsb_release -is'
2016-06-13T12:06:17.298 DEBUG:teuthology.misc:System to be installed: CentOS
2016-06-13T12:06:17.298 INFO:teuthology.packaging:Looking for package version: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/082f918a99807af297f75b7eb0fda627e24725f1/version
2016-06-13T12:06:17.303 INFO:teuthology.packaging:Package found...
2016-06-13T12:06:17.303 INFO:teuthology.packaging:Found version: 10.2.0
2016-06-13T12:06:17.304 INFO:teuthology.orchestra.run.vpm115:Running: "rpm -q ceph --qf '%{VERSION}'" 
2016-06-13T12:06:17.384 INFO:teuthology.packaging:The installed version of ceph is 10.2.1
2016-06-13T12:06:17.384 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1020, in upgrade
    upgrade_common(ctx, config, upgrade_old_style)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 967, in upgrade_common
149951499514995    verify_package_version(ctx, node, remote)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 287, in verify_package_version
    pkg=pkg_to_check
RuntimeError: ceph version 10.2.0 was not installed, found 10.2.1.

and on ubuntu:
Job: 256903
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-13_11:15:49-upgrade:jewel-x-master-distro-basic-vps/256903/teuthology.log

2016-06-13T11:50:10.281 INFO:teuthology.orchestra.run.vpm011.stdout:Reading package lists...
2016-06-13T11:50:10.285 INFO:teuthology.orchestra.run.vpm011:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=10.2.0-2260-g082f918-1trusty rbd-fuse=10.2.0-2260-g082f918-1trusty librbd1=10.2.0-2260-g082f918-1trusty ceph-fuse=10.2.0-2260-g082f918-1trusty python-ceph=10.2.0-2260-g082f918-1trusty ceph-common=10.2.0-2260-g082f918-1trusty libcephfs-java=10.2.0-2260-g082f918-1trusty ceph=10.2.0-2260-g082f918-1trusty libcephfs-jni=10.2.0-2260-g082f918-1trusty ceph-test=10.2.0-2260-g082f918-1trusty radosgw=10.2.0-2260-g082f918-1trusty librados2=10.2.0-2260-g082f918-1trusty libcephfs1=10.2.0-2260-g082f918-1trusty'
2016-06-13T11:50:10.307 INFO:teuthology.orchestra.run.vpm011.stdout:Reading package lists...
2016-06-13T11:50:10.469 INFO:teuthology.orchestra.run.vpm011.stdout:Building dependency tree...
2016-06-13T11:50:10.470 INFO:teuthology.orchestra.run.vpm011.stdout:Reading state information...
2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:Some packages could not be installed. This may mean that you have
2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:requested an impossible situation or if you are using the unstable
2016-06-13T11:50:10.585 INFO:teuthology.orchestra.run.vpm011.stdout:distribution that some required packages have not yet been created
2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout:or been moved out of Incoming.
2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout:The following information may help to resolve the situation:
2016-06-13T11:50:10.586 INFO:teuthology.orchestra.run.vpm011.stdout:
2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout:The following packages have unmet dependencies:
2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout: ceph-common : Depends: librbd1 (= 10.2.1-241-g1720664-1trusty) but 10.2.0-2260-g082f918-1trusty is to be installed
2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: libradosstriper1 but it is not going to be installed
2016-06-13T11:50:10.587 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: librgw2 but it is not going to be installed
2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: python-rados (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2149951499514995016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: python-cephfs (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: python-rbd (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.588 INFO:teuthology.orchestra.run.vpm011.stdout: ceph-test : Depends: libradosstriper1 but it is not going to be installed
2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout:             Depends: librgw2 but it is not going to be installed
2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout: python-ceph : Depends: python-rados (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.589 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: python-rbd (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.590 INFO:teuthology.orchestra.run.vpm011.stdout:               Depends: python-cephfs (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.590 INFO:teuthology.orchestra.run.vpm011.stdout: radosgw : Depends: librgw2 (= 10.2.1-241-g1720664-1trusty) but it is not going to be installed
2016-06-13T11:50:10.601 INFO:teuthology.orchestra.run.vpm011.stderr:E: Unable to correct problems, you have held broken packages.

#15 Updated by Nathan Cutler about 3 years ago

  • Status changed from New to Verified
  • Assignee set to Nathan Cutler

#16 Updated by Nathan Cutler about 3 years ago

I think these problems are caused by tags being present in one branch, but absent in another, combined with the use of "git describe" to determine the package (rpm, deb) version.

Consider the following example:

First, I install jewel 10.2.1. The package version string starts with "10.2.1". This is derived from the output of "git describe". For example, in comment #14995-14 above we have ceph packages with version string "10.2.1-241-g1720664-1trusty". That means the packages were built from SHA1 1720664 which is 241 commits ahead of tag v10.2.1. Please note that the "v10.2.1" tag exists only in the jewel branch.

Next, I attempt to upgrade to master, which has a "v10.2.0" tag but no "v10.2.1" tag and indeed "v10.2.0" is the latest tag in master. As a result, "git describe" on current master returns:

v10.2.0-2846-g5de4a9c

And of course gitbuilder packages built from the master branch have a version that starts with "10.2.0" and the package managers will refuse to upgrade from "10.2.1" to "10.2.0".

The jewel -> master upgrade scenario will work as long as "v10.2.0" is the highest tag in both branches. Once jewel gets a "v10.2.1" tag, an upgrade from jewel to master will be impossible until master gets a v11.0.0 tag.

I suspect all instances of this bug are caused by variations on this theme.

As a fix, a sanity check could be added at the beginning of the upgrade suite. It would assert that the destination version string is greater than the source version string. If this fails, the suite would fail with a clear error right at the beginning.

#17 Updated by Nathan Cutler about 3 years ago

  • Status changed from Verified to Resolved

Sage added the v11.0.0 tag to master and this tag will show up in kraken, too, when master is merged into kraken.

I opened #16521 for adding the pre-upgrade sanity check.

Also available in: Atom PDF