Project

General

Profile

Actions

Bug #24105

closed

"Command failed on smithi005 with status 1: "sudo yum -y install '' libcephfs_jni1"" in client upgrades

Added by Yuri Weinstein almost 6 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/client-upgrade
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is for:

upgrade:client-upgrade-luminous-mimic
upgrade:client-upgrade-jewel-mimic
upgrade:client-upgrade-hammer-mimic
(upgrade:client-upgrade-kraken-mimic actually works fine !)

Runs: http://pulpito.ceph.com/teuthology-2018-05-12_14:15:07-upgrade:client-upgrade-luminous-mimic-distro-basic-smithi/
http://pulpito.ceph.com/teuthology-2018-05-12_14:13:44-upgrade:client-upgrade-jewel-mimic-distro-basic-smithi/
http://pulpito.ceph.com/teuthology-2018-05-12_14:13:00-upgrade:client-upgrade-hammer-mimic-distro-basic-smithi/

Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2018-05-12_14:15:07-upgrade:client-upgrade-luminous-mimic-distro-basic-smithi/2527416/teuthology.log

2018-05-12T14:40:15.474 INFO:teuthology.packaging:branch: luminous
2018-05-12T14:40:15.474 INFO:teuthology.packaging:sha1: 1af66d54f173f933d6a26ffb617b429f2c700ac6
2018-05-12T14:40:15.474 INFO:teuthology.orchestra.run.smithi079:Running: "if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\\ngpg/enabled=1\\npriority=1\\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/luminous/;g' /etc/yum.repos.d/ceph.repo ; fi" 
2018-05-12T14:40:15.507 INFO:teuthology.orchestra.run.smithi005:Running: 'test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig'
2018-05-12T14:40:15.576 INFO:teuthology.orchestra.run.smithi079:Running: 'test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig'
2018-05-12T14:40:15.593 INFO:teuthology.orchestra.run.smithi005:Running: "grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf" 
2018-05-12T14:40:15.659 INFO:teuthology.orchestra.run.smithi079:Running: "grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf" 
2018-05-12T14:40:15.663 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1
2018-05-12T14:40:15.664 INFO:teuthology.orchestra.run.smithi005:Running: 'sudo yum clean all'
2018-05-12T14:40:15.741 INFO:teuthology.orchestra.run.smithi079.stdout:check_obsoletes = 1
2018-05-12T14:40:15.742 INFO:teuthology.orchestra.run.smithi079:Running: 'sudo yum clean all'
2018-05-12T14:40:15.874 INFO:teuthology.orchestra.run.smithi005.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2018-05-12T14:40:15.917 INFO:teuthology.orchestra.run.smithi005.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras
2018-05-12T14:40:15.918 INFO:teuthology.orchestra.run.smithi005.stdout:              : lab-extras updates
2018-05-12T14:40:15.918 INFO:teuthology.orchestra.run.smithi005.stdout:Cleaning up everything
2018-05-12T14:40:15.918 INFO:teuthology.orchestra.run.smithi005.stdout:Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
2018-05-12T14:40:15.926 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2018-05-12T14:40:15.964 INFO:teuthology.orchestra.run.smithi079.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras
2018-05-12T14:40:15.964 INFO:teuthology.orchestra.run.smithi079.stdout:              : lab-extras updates
2018-05-12T14:40:15.964 INFO:teuthology.orchestra.run.smithi079.stdout:Cleaning up everything
2018-05-12T14:40:15.964 INFO:teuthology.orchestra.run.smithi079.stdout:Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
2018-05-12T14:40:15.971 INFO:teuthology.orchestra.run.smithi005.stdout:Cleaning up list of fastest mirrors
2018-05-12T14:40:15.992 INFO:teuthology.orchestra.run.smithi005:Running: "sudo yum -y install '' libcephfs_jni1" 
2018-05-12T14:40:16.000 INFO:teuthology.orchestra.run.smithi079.stdout:Cleaning up list of fastest mirrors
2018-05-12T14:40:16.013 INFO:teuthology.orchestra.run.smithi079:Running: "sudo yum -y install '' libcephfs_jni1" 
2018-05-12T14:40:16.161 INFO:teuthology.orchestra.run.smithi079.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2018-05-12T14:40:16.220 INFO:teuthology.orchestra.run.smithi005.stdout:Loaded plugins: fastestmirror, langpacks, priorities
2018-05-12T14:40:17.444 INFO:teuthology.orchestra.run.smithi005.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/f2f5ddd4101a588a7c6eb60e20c45756b87e8123994fe1b3f0116a57ef74ce89-comps-Everything.x86_64.xml.gz: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:17.445 INFO:teuthology.orchestra.run.smithi005.stderr:Trying other mirror.
..................
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:failure: repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:http://mirror.pnl.gov/epel/7/x86_64/repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:24.575 INFO:teuthology.orchestra.run.smithi005.stderr:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/43cc8c9e36e1d245e92227c20d47a5be24f06232b7a7d174dff7da8804065351-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
2018-05-12T14:40:24.609 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 250, in install
    install_packages(ctx, package_list, config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 123, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/rpm.py", line 153, in _update_package_list_and_install
    pkg_mng_subcommand_opts, cpack
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 423, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 155, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 177, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi005 with status 1: "sudo yum -y install '' libcephfs_jni1" 
Actions #1

Updated by Greg Farnum almost 6 years ago

  • Status changed from New to Closed

Kinda looks like the networking flaked out while it was trying to install packages.

Actions #2

Updated by David Galloway almost 6 years ago

This is epel mirror failure. The primary epel repo got corrupted and the mirrors synced the badness.

https://www.spinics.net/linux/fedora/epel-devel/msg02055.html

I gave Yuri a yaml fragment to pull directly from fedoraproject.org

Actions

Also available in: Atom PDF