Project

General

Profile

Actions
Copy link

Bug #18492

open

upgrade:hammer-jewel-x fails on xenial b/c of different packages

Added by Yuri Weinstein over 7 years ago. Updated about 7 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2017-01-10_23:01:29-upgrade:hammer-jewel-x-kraken-distro-basic-smithi/707243/teuthology.log

CommandFailedError: Command failed on smithi088 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-fs-common=0.94.9-4511-g48687c7-1xenial ceph-fuse=0.94.9-4511-g48687c7-1xenial python-rados=0.94.9-4511-g48687c7-1xenial ceph=0.94.9-4511-g48687c7-1xenial ceph-test=0.94.9-4511-g48687c7-1xenial rbd-mirror=0.94.9-4511-g48687c7-1xenial rbd-nbd=0.94.9-4511-g48687c7-1xenial librbd-dev=0.94.9-4511-g48687c7-1xenial libradosstriper1=0.94.9-4511-g48687c7-1xenial rbd-fuCommandFailedError: Command failed on smithi088 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-fs-common=0.94.9-4511-g48687c7-1xenial ceph-fuse=0.94.9-4511-g48687c7-1xenial python-rados=0.94.9-4511-g48687c7-1xenial ceph=0.94.9-4511-g48687c7-1xenial ceph-test=0.94.9-4511-g48687c7-1xenial rbd-mirror=0.94.9-4511-g48687c7-1xenial rbd-nbd=0.94.9-4511-g48687c7-1xenial librbd-dev=0.94.9-4511-g48687c7-1xenial libradosstriper1=0.94.9-4511-g48687c7-1xenial rbd-fuse=0.94.9-4511-g48687c7-1xenial libcephfs-jni=0.94.9-4511-g48687c7-1xenial radosgw=0.94.9-4511-g48687c7-1xenial librados2=0.94.CommandFailedError: Command failed on smithi088 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-fs-common=0.94.9-4511-g48687c7-1xenial ceph-fuse=0.94.9-4511-g48687c7-1xenial python-rados=0.94.9-4511-g48687c7-1xenial ceph=0.94.9-4511-g48687c7-1xenial ceph-test=0.94.9-4511-g48687c7-1xenial rbd-mirror=0.94.9-4511-g48687c7-1xenial rbd-nbd=0.94.9-4511-g48687c7-1xenial librbd-dev=0.94.9-4511-g48687c7-1xenial libradosstriper1=0.94.9-4511-g48687c7-1xenial rbd-fuse=0.94.9-4511-g48687c7-1xenial libcephfs-jni=0.94.9-4511-g48687c7-1xenial radosgw=0.94.9-4511-g48687c7-1xenial librados2=0.94.9-4511-g48687c7-1xenial ceph-mon=0.94.9-4511-g48687c7-1xenial libcephfs1=0.94.9-4511-g48687c7-1xenial librgw2=0.94.9-4511-g48687c7-1xenial ceph-mds=0.94.9-4511-g48687c7-1xenial librbd1=0.94.9-4511-g48687c7-1xenial python-ceph=0.94.9-4511-g48687c7-1xenial ceph-osd=0.94.9-4511-g48687c7-1xenial ceph-base=0.94.9-4511-g48687c7-1xenial python-cephfs=0.94.9-4511-g48687c7-1xenial python-rbd=0.94.9-4511-g48687c7-1xenial ceph-common=0.94.9-4511-g48687c7-1xenial libcephfs-java=0.94.9-4511-g48687c7-1xenial'9-4511-g48687c7-1xenial ceph-mon=0.94.9-4511-g48687c7-1xenial libcephfs1=0.94.9-4511-g48687c7-1xenial librgw2=0.94.9-4511-g48687c7-1xenial ceph-mds=0.94.9-4511-g48687c7-1xenial librbd1=0.94.9-4511-g48687c7-1xenial python-ceph=0.94.9-4511-g48687c7-1xenial ceph-osd=0.94.9-4511-g48687c7-1xenial ceph-base=0.94.9-4511-g48687c7-1xenial python-cephfs=0.94.9-4511-g48687c7-1xenial python-rbd=0.94.9-4511-g48687c7-1xenial ceph-common=0.94.9-4511-g48687c7-1xenial libcephfs-java=0.94.9-4511-g48687c7-1xenial'se=0.94.9-4511-g48687c7-1xenial libcephfs-jni=0.94.9-4511-g48687c7-1xenial radosgw=0.94.9-4511-g48687c7-1xenial librados2=0.94.9-4511-g48687c7-1xenial ceph-mon=0.94.9-4511-g48687c7-1xenial libcephfs1=0.94.9-4511-g48687c7-1xenial librgw2=0.94.9-4511-g48687c7-1xenial ceph-mds=0.94.9-4511-g48687c7-1xenial librbd1=0.94.9-4511-g48687c7-1xenial python-ceph=0.94.9-4511-g48687c7-1xenial ceph-osd=0.94.9-4511-g48687c7-1xenial ceph-base=0.94.9-4511-g48687c7-1xenial python-cephfs=0.94.9-4511-g48687c7-1xenial python-rbd=0.94.9-4511-g48687c7-1xenial ceph-common=0.94.9-4511-g48687c7-1xenial libcephfs-java=0.94.9-4511-g48687c7-1xenial'

Per troubleshooting with Josh the reason is "hammer and jewel + later use different packages"

Actions
Copy link
 #1

Updated by Josh Durgin over 7 years ago

packages like rbd-mirror, ceph-mgr, etc. don't exist in hammer. We need to mention them explicitly on xenial to be able to install older packages at all (since xenial ships 10.2.2) - e.g. https://github.com/jdurgin/ceph/commit/2932c3b248ae8bfb356a33da59a06da6e0163d0c.

For ugrade tests using hammer, we'll need a way to use one set of packages for hammer and another jewel, etc. Perhaps making install.upgrade use a packages.yaml in a specific branch would work. The caveat I see with that is for xenial, needing to replace all the existing ceph packages to avoid dependency issues.

Actions
Copy link
 #2

Updated by Dan Mick about 7 years ago

Research:

hammer: (0.94.x):
ceph
ceph-common
ceph-common-dbg
ceph-dbg
ceph-fs-common
ceph-fs-common-dbg
ceph-fuse
ceph-fuse-dbg
ceph-mds
ceph-mds-dbg
ceph-resource-agents
ceph-test
ceph-test-dbg
libcephfs1
libcephfs1-dbg
libcephfs-dev
libcephfs-java
libcephfs-jni
librados2
librados2-dbg
librados-dev
libradosstriper1
libradosstriper1-dbg
libradosstriper-dev
librbd1
librbd1-dbg
librbd-dev
python-ceph
python-cephfs
python-rados
python-rbd
radosgw
radosgw-dbg
rbd-fuse
rbd-fuse-dbg
rest-bench
rest-bench-dbg

jewel (10.2.x):
ceph
ceph-base
ceph-common
ceph-common-dbg
ceph-fs-common
ceph-fs-common-dbg
ceph-fuse
ceph-fuse-dbg
ceph-mds
ceph-mds-dbg
ceph-mon
ceph-mon-dbg
ceph-osd
ceph-osd-dbg
ceph-resource-agents
ceph-test
ceph-test-dbg
libcephfs1
libcephfs1-dbg
libcephfs-dev
libcephfs-java
libcephfs-jni
librados2
librados2-dbg
librados-dev
libradosstriper1
libradosstriper1-dbg
libradosstriper-dev
librbd1
librbd1-dbg
librbd-dev
librgw2
librgw2-dbg
librgw-dev
python-ceph
python-cephfs
python-rados
python-rbd
radosgw
radosgw-dbg
rbd-fuse
rbd-fuse-dbg
rbd-mirror
rbd-mirror-dbg
rbd-nbd
rbd-nbd-dbg

Differences:
--- v0.94.10.pkgs 2017-02-22 16:05:51.104025728 -0800
+++ v10.1.0.pkgs 2017-02-22 16:02:56.727218330 -0800
@ -1,13 +1,17 @
ceph
+ceph-base
ceph-common
ceph-common-dbg
-ceph-dbg
ceph-fs-common
ceph-fs-common-dbg
ceph-fuse
ceph-fuse-dbg
ceph-mds
ceph-mds-dbg
+ceph-mon
+ceph-mon-dbg
+ceph-osd
+ceph-osd-dbg
ceph-resource-agents
ceph-test
ceph-test-dbg
@ -25,6 +29,9 @
librbd1
librbd1-dbg
librbd-dev
+librgw2
+librgw2-dbg
+librgw-dev
python-ceph
python-cephfs
python-rados
@ -33,5 +40,7 @
radosgw-dbg
rbd-fuse
rbd-fuse-dbg
-rest-bench
-rest-bench-dbg
+rbd-mirror
+rbd-mirror-dbg
+rbd-nbd
+rbd-nbd-dbg

Actions
Copy link
 #4

Updated by Dan Mick about 7 years ago

So I think there are several problems.

1) The original log quoted here seems to be suffering from "the package list for install refers to packages that did not exist in hammer". This could be solved by adding package overrides to the suite configuration with complete lists of packages, as documented in the install task:

    overrides:
      install:
        ceph:
          packages:
            deb:
            - ceph-osd
            - ceph-mon
            rpm:
            - ceph-devel
            - rbd-fuse

2) installing older releases on distros that have newer releases in the distro repos, I think, can be fixed by making certain that all ceph packages are uninstalled. this may involve:

2a) making nuke use a package list that includes the union of all ceph package names ever used (and allowing removal to fail if the package name isn't known to the available repos)

2b) making sure that every installation explicitly specifies a complete list of ceph packages for the release that's being installed.

2a) will have to involve a new package list from somewhere, and it will have to address the problem of other non-ceph packages that depend on ceph packages (like qemu-kvm).

I think 2b) ought be achievable by making sure packages.yaml is correct and complete for every distinguished branch of ceph-qa-suite.git and ceph.git, and it should also solve 1).

Actions
Copy link
 #5

Updated by Dan Mick about 7 years ago

  • Subject changed from upgrade:hammer-jewel-x failes on xenial b/c of different packages to upgrade:hammer-jewel-x fails on xenial b/c of different packages
Actions
Copy link

Also available in: Atom PDF