Project

General

Profile

Actions
Copy link

Bug #21562

closed

package list needs fixing in upgrade:client-upgrade-luminous

Added by Yuri Weinstein over 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/client-upgrade
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2017-09-21_05:45:02-upgrade:client-upgrade-luminous-distro-basic-smithi/
Jobs: ['1654693', '1654688', '1654691']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-09-21_05:45:02-upgrade:client-upgrade-luminous-distro-basic-smithi/1654688/teuthology.log

2017-09-24T13:05:10.192 INFO:teuthology.orchestra.run.smithi096.stdout:Fetched 322 kB in 0s (470 kB/s)
2017-09-24T13:05:11.249 INFO:teuthology.orchestra.run.smithi096.stdout:Reading package lists...
2017-09-24T13:05:11.276 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists...
2017-09-24T13:05:11.280 INFO:teuthology.orchestra.run.smithi096:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial'
2017-09-24T13:05:11.312 INFO:teuthology.orchestra.run.smithi194:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial'
2017-09-24T13:05:11.349 INFO:teuthology.orchestra.run.smithi096.stdout:Reading package lists...
2017-09-24T13:05:11.381 INFO:teuthology.orchestra.run.smithi194.stdout:Reading package lists...
2017-09-24T13:05:11.601 INFO:teuthology.orchestra.run.smithi096.stdout:Building dependency tree...
2017-09-24T13:05:11.601 INFO:teuthology.orchestra.run.smithi096.stdout:Reading state information...
2017-09-24T13:05:11.631 INFO:teuthology.orchestra.run.smithi194.stdout:Building dependency tree...
2017-09-24T13:05:11.631 INFO:teuthology.orchestra.run.smithi194.stdout:Reading state information...
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:Some packages could not be installed. This may mean that you have
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:requested an impossible situation or if you are using the unstable
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:distribution that some required packages have not yet been created
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:or been moved out of Incoming.
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:The following information may help to resolve the situation:
2017-09-24T13:05:11.670 INFO:teuthology.orchestra.run.smithi096.stdout:
2017-09-24T13:05:11.671 INFO:teuthology.orchestra.run.smithi096.stdout:The following packages have unmet dependencies:
2017-09-24T13:05:11.697 INFO:teuthology.orchestra.run.smithi194.stdout:Some packages could not be installed. This may mean that you have
2017-09-24T13:05:11.697 INFO:teuthology.orchestra.run.smithi194.stdout:requested an impossible situation or if you are using the unstable
2017-09-24T13:05:11.697 INFO:teuthology.orchestra.run.smithi194.stdout:distribution that some required packages have not yet been created
2017-09-24T13:05:11.698 INFO:teuthology.orchestra.run.smithi194.stdout:or been moved out of Incoming.
2017-09-24T13:05:11.698 INFO:teuthology.orchestra.run.smithi194.stdout:The following information may help to resolve the situation:
2017-09-24T13:05:11.699 INFO:teuthology.orchestra.run.smithi194.stdout:
2017-09-24T13:05:11.699 INFO:teuthology.orchestra.run.smithi194.stdout:The following packages have unmet dependencies:
2017-09-24T13:05:11.709 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common : Depends: python-rados (= 0.94.10-1xenial) but it is not going to be installed
2017-09-24T13:05:11.710 INFO:teuthology.orchestra.run.smithi096.stdout:               Depends: python-cephfs (= 0.94.10-1xenial) but 10.2.7-0ubuntu0.16.04.1 is to be installed
2017-09-24T13:05:11.710 INFO:teuthology.orchestra.run.smithi096.stdout:               Depends: python-rbd (= 0.94.10-1xenial) but it is not going to be installed
2017-09-24T13:05:11.710 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test : Depends: libradosstriper1 but it is not going to be installed
2017-09-24T13:05:11.712 INFO:teuthology.orchestra.run.smithi096.stdout: python-ceph : Depends: python-rados but it is not going to be installed
2017-09-24T13:05:11.712 INFO:teuthology.orchestra.run.smithi096.stdout:               Depends: python-rbd but it is not going to be installed
2017-09-24T13:05:11.724 INFO:teuthology.orchestra.run.smithi096.stderr:E: Unable to correct problems, you have held broken packages.
2017-09-24T13:05:11.727 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 246, in install
    install_packages(ctx, package_list, config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 119, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/deb.py", line 67, in _update_package_list_and_install
    ] + ['%s=%s' % (d, version) for d in debs],
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 423, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 155, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 177, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi096 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial'
Actions
Copy link
 #1

Updated by David Galloway over 6 years ago

This package list works:

ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial python-cephfs=0.94.10-1xenial python-rados=0.94.10-1xenial python-rbd=0.94.10-1xenial libradosstriper1=0.94.10-1xenial

Doesn't work:

ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial 
ubuntu@vpm181:~$ sudo DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=0.94.10-1xenial rbd-fuse=0.94.10-1xenial librbd1=0.94.10-1xenial ceph-fuse=0.94.10-1xenial python-ceph=0.94.10-1xenial ceph-common=0.94.10-1xenial libcephfs-java=0.94.10-1xenial ceph=0.94.10-1xenial libcephfs-jni=0.94.10-1xenial ceph-test=0.94.10-1xenial radosgw=0.94.10-1xenial librados2=0.94.10-1xenial python-cephfs=0.94.10-1xenial python-rados=0.94.10-1xenial python-rbd=0.94.10-1xenial libradosstriper1=0.94.10-1xenial
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  binutils ceph-fs-common javascript-common libaio1 libbabeltrace-ctf1 libbabeltrace1 libboost-iostreams1.58.0 libboost-program-options1.58.0 libboost-random1.58.0 libboost-system1.58.0 libboost-thread1.58.0 libcephfs1 libfcgi0ldbl
  libgoogle-perftools4 libjs-jquery libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libnspr4 libnss3 libnss3-nssdb libsnappy1v5 libtcmalloc-minimal4 libunwind8 liburcu4 python-blinker python-cffi-backend python-chardet
  python-cryptography python-enum34 python-flask python-idna python-ipaddress python-itsdangerous python-jinja2 python-markupsafe python-ndg-httpsclient python-openssl python-pkg-resources python-pyasn1 python-pyinotify python-requests
  python-six python-urllib3 python-werkzeug xml2
Suggested packages:
  binutils-doc apache2 | lighttpd | httpd python-blinker-doc python-cryptography-doc python-cryptography-vectors python-enum34-doc python-flask-doc python-jinja2-doc python-openssl-doc python-openssl-dbg python-setuptools doc-base
  python-pyinotify-doc python-ntlm ipython python-genshi python-lxml python-greenlet python-redis python-pylibmc | python-memcache python-werkzeug-doc
The following NEW packages will be installed:
  binutils ceph ceph-common ceph-fs-common ceph-fuse ceph-mds ceph-test javascript-common libaio1 libbabeltrace-ctf1 libbabeltrace1 libboost-iostreams1.58.0 libboost-program-options1.58.0 libboost-random1.58.0 libboost-system1.58.0
  libboost-thread1.58.0 libcephfs-java libcephfs-jni libcephfs1 libfcgi0ldbl libgoogle-perftools4 libjs-jquery libleveldb1v5 liblttng-ust-ctl2 liblttng-ust0 libnspr4 libnss3 libnss3-nssdb librados2 libradosstriper1 librbd1 libsnappy1v5
  libtcmalloc-minimal4 libunwind8 liburcu4 python-blinker python-ceph python-cephfs python-cffi-backend python-chardet python-cryptography python-enum34 python-flask python-idna python-ipaddress python-itsdangerous python-jinja2
  python-markupsafe python-ndg-httpsclient python-openssl python-pkg-resources python-pyasn1 python-pyinotify python-rados python-rbd python-requests python-six python-urllib3 python-werkzeug radosgw rbd-fuse xml2
0 upgraded, 62 newly installed, 0 to remove and 186 not upgraded.
Need to get 80.9 MB of archives.
After this operation, 392 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
Actions
Copy link
 #2

Updated by David Galloway over 6 years ago

This works:

root@vpm181:~# cat /etc/apt/preferences.d/ceph 
Package: *
Pin: origin *.ceph.com
Pin-Priority: 900

It might not be a bad idea to have teuthology write that file. We'd likely never see this again.

The problem is some of the packages in the package list have dependencies that the Ubuntu repo provides. Except the packages in the Ubuntu repo are newer than the ones we actually want to install. When we tell apt to use the chacra hosts (by using a wildcard *.ceph.com) at a higher priority, dependency resolution works.

https://manpages.debian.org/stretch/apt/apt_preferences.5.en.html#How_APT_Interprets_Priorities

Actions
Copy link
 #3

Updated by David Galloway over 6 years ago

Derp, we were already doing this in ceph-cm-ansible. The file is currently set to use gitbuilder.ceph.com though.

@Yuri, re-run that failing test with ceph-cm-ansible branch: wip-apt-pinning

Actions
Copy link
 #5

Updated by Sage Weil about 6 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF