Bug #10310: Error "'sudo yum install ceph-devel-0.XXX -y'" across nighties runs - devops - Ceph

Bug #10310

Error "'sudo yum install ceph-devel-0.XXX -y'" across nighties runs

Added by Yuri Weinstein almost 9 years ago. Updated about 7 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Nathan Cutler

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

In Octo lab several jobs failed.

Logs are in http://magna002.ceph.redhat.com/teuthology-2014-12-13_12:33:46-rados-dumpling-distro-basic-magna/3394/ for one

2014-12-13T12:49:09.050 INFO:teuthology.orchestra.run.magna088.stderr:Error: Package: 1:ceph-common-0.80.7-2.el7.x86_64 (download.devel.redhat.com_rel-eng_RHEL-7.1-20141204.2_compose_Server_x86_64_os_)
2014-12-13T12:49:09.051 INFO:teuthology.orchestra.run.magna088.stderr:           Requires: librbd1 = 1:0.80.7-2.el7
2014-12-13T12:49:09.090 INFO:teuthology.orchestra.run.magna088.stderr:           Installed: librbd1-0.67.11-60.g9bca077.el7.x86_64 (@Ceph)
2014-12-13T12:49:09.091 INFO:teuthology.orchestra.run.magna088.stderr:               librbd1 = 0.67.11-60.g9bca077.el7
2014-12-13T12:49:09.092 INFO:teuthology.orchestra.run.magna088.stderr:Error: Package: 1:python-rbd-0.80.7-2.el7.x86_64 (download.devel.redhat.com_rel-eng_RHEL-7.1-20141204.2_compose_Server_x86_64_os_)
2014-12-13T12:49:09.093 INFO:teuthology.orchestra.run.magna088.stderr:           Requires: librbd1 = 1:0.80.7-2.el7
2014-12-13T12:49:09.094 INFO:teuthology.orchestra.run.magna088.stderr:           Installed: librbd1-0.67.11-60.g9bca077.el7.x86_64 (@Ceph)
2014-12-13T12:49:09.095 INFO:teuthology.orchestra.run.magna088.stderr:               librbd1 = 0.67.11-60.g9bca077.el7
2014-12-13T12:49:09.096 INFO:teuthology.orchestra.run.magna088.stderr:Error: Package: 1:ceph-common-0.80.7-2.el7.x86_64 (download.devel.redhat.com_rel-eng_RHEL-7.1-20141204.2_compose_Server_x86_64_os_)
2014-12-13T12:49:09.097 INFO:teuthology.orchestra.run.magna088.stderr:           Requires: librados2 = 1:0.80.7-2.el7
2014-12-13T12:49:09.098 INFO:teuthology.orchestra.run.magna088.stderr:           Installed: librados2-0.67.11-60.g9bca077.el7.x86_64 (@Ceph)
2014-12-13T12:49:09.099 INFO:teuthology.orchestra.run.magna088.stderr:               librados2 = 0.67.11-60.g9bca077.el7
2014-12-13T12:49:09.100 INFO:teuthology.orchestra.run.magna088.stderr:Error: Package: 1:python-rados-0.80.7-2.el7.x86_64 (download.devel.redhat.com_rel-eng_RHEL-7.1-20141204.2_compose_Server_x86_64_os_)
2014-12-13T12:49:09.101 INFO:teuthology.orchestra.run.magna088.stderr:           Requires: librados2 = 1:0.80.7-2.el7
2014-12-13T12:49:09.102 INFO:teuthology.orchestra.run.magna088.stderr:           Installed: librados2-0.67.11-60.g9bca077.el7.x86_64 (@Ceph)
2014-12-13T12:49:09.103 INFO:teuthology.orchestra.run.magna088.stderr:               librados2 = 0.67.11-60.g9bca077.el7
2014-12-13T12:49:09.104 INFO:teuthology.orchestra.run.magna088.stdout: You could try using --skip-broken to work around the problem
2014-12-13T12:49:09.933 INFO:teuthology.orchestra.run.magna088.stdout: You could try running: rpm -Va --nofiles --nodigest
2014-12-13T12:49:10.074 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 494, in _update_rpm_package_list_and_install
    remote.run(args=['sudo', 'yum', 'install', pkg2add, '-y'])
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on magna088 with status 1: 'sudo yum install ceph-devel-0.67.11 -y'
2014-12-13T12:49:10.077 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 808, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 558, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 494, in _update_rpm_package_list_and_install
    remote.run(args=['sudo', 'yum', 'install', pkg2add, '-y'])
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on magna088 with status 1: 'sudo yum install ceph-devel-0.67.11 -y'
2014-12-13T12:49:10.079 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1280, in task
    lambda: ship_utilities(ctx=ctx, config=None),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 808, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 558, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 494, in _update_rpm_package_list_and_install
    remote.run(args=['sudo', 'yum', 'install', pkg2add, '-y'])
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on magna088 with status 1: 'sudo yum install ceph-devel-0.67.11 -y'

History

#1 Updated by Ken Dreyer almost 9 years ago

In RHEL 6.7 and RHEL 7.1, the librados and librbd packages (specifically version 0.80.7) are included in the RHEL Base repositories.

So, under the assumption that we want to continue testing Dumpling on RHEL 6 and 7, we should set the ceph.com repositories to have a higher yum priority than the RHEL Base repositories.

Long term, we should evaluate the feasibility of continuing to test Dumpling on RHEL 6 and 7, since I'd be surprised if users want to stay on such an old version. This issue is going to bite all Dumpling-on-RHEL users when RHEL 7.1 and 6.7 ship.

#2 Updated by Zack Cerza almost 9 years ago

Assignee changed from Sandon Van Ness to Warren Usui

#3 Updated by Warren Usui almost 9 years ago

This is caused by the same problem that caused #10308

#4 Updated by Yuri Weinstein over 8 years ago

Priority changed from Normal to Urgent

#5 Updated by Yuri Weinstein over 8 years ago

Same set of problems in this run http://pulpito.ceph.redhat.com/teuthology-2015-01-06_22:00:01-rados-dumpling-distro-basic-magna/

#6 Updated by Warren Usui over 8 years ago

sudo yum install ceph-devel-0.67.11 -y --> I am wondering if there is a version issue here. This returns an error code of 1 when run on one of the magna machines. sudo yum install ceph-0.80.5 works. Same with ceph-radosgw.

I will see if I can find another rhel version that works.

#7 Updated by Yuri Weinstein over 8 years ago

Subject changed from Error "'sudo yum install ceph-devel-0.67.11 -y'" in rados-dumpling-distro-basic-magna run to Error "'sudo yum install ceph-devel-0.XXX -y'" across nighties runs

In run http://pulpito.ceph.redhat.com/teuthology-2015-01-07_02:10:01-samba-firefly-distro-basic-magna/

We get 'sudo yum install ceph-radosgw-0.80.7 -y'

Logs are in http://magna002.ceph.redhat.com/teuthology-2015-01-07_02:10:01-samba-firefly-distro-basic-magna/21790/

#8 Updated by Ken Dreyer over 8 years ago

Yuri Weinstein wrote:

In run http://pulpito.ceph.redhat.com/teuthology-2015-01-07_02:10:01-samba-firefly-distro-basic-magna/

We get 'sudo yum install ceph-radosgw-0.80.7 -y'

Logs are in http://magna002.ceph.redhat.com/teuthology-2015-01-07_02:10:01-samba-firefly-distro-basic-magna/21790/

The problem with that host is that there was an old dumpling release installed on that box at the time that the test tried to to run. How did that happen?

Can you retry that particular one on a completely re-imaged system?

I didn't look in-depth at the other yum-related test run failures, but I suspect that if you tried the runs on completely re-imaged systems, the issues might go away.

#9 Updated by Sage Weil over 8 years ago

Priority changed from Urgent to Immediate

#10 Updated by Yuri Weinstein over 8 years ago

Ken, you maybe right, but we need to address this issue from high level standpoint, so those issues causing lots of suites failed - resolved.

#11 Updated by Yuri Weinstein over 8 years ago

Assignee changed from Warren Usui to Sandon Van Ness

#12 Updated by Sage Weil over 8 years ago

Status changed from New to Resolved

#13 Updated by Zack Cerza over 8 years ago

How was this fixed?

#14 Updated by J. Eric Ivancich about 7 years ago

I don't know if this is the same bug or different.
A teuthology job errored out due to:

CommandFailedError: Command failed on mira054 with status 1: 'sudo yum install ceph-devel -y'

I re-ran the exact same job and it passed.

Failure: http://pulpito.ceph.com/ivancich-2016-09-03_16:32:36-dmclock:rados:basic-wip_dmc2_segfault1---basic-mira

Pass: http://pulpito.ceph.com/ivancich-2016-09-03_16:54:35-dmclock:rados:basic-wip_dmc2_segfault1---basic-mira/

More of error:

CommandFailedError: Command failed on mira054 with status 1: 'sudo yum install ceph-devel -y'
2016-09-03T16:40:37.041 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=0eba0c7a312e44fdbcf0cde4522c0b5f
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 69, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1289, in task
    lambda: ship_utilities(ctx=ctx, config=None),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 626, in install
    install_packages(ctx, package_list, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 361, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 261, in _update_rpm_package_list_and_install
    remote.run(args=['sudo', 'yum', 'install', cpack, '-y'])
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 399, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)

#15 Updated by Nathan Cutler about 7 years ago

ceph-devel has been dropped in master, yet the suite branch wip_dmclock2 is asking to install it:

$ git checkout wip_dmclock2
Already on 'wip_dmclock2'
Your branch is up-to-date with 'ceph/wip_dmclock2'.
$ grep -r 'ceph-devel'
packages/packages.yaml:  - ceph-devel
$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ grep -r 'ceph-devel'
$

So, just drop ceph-devel from packages.yaml and you'll be good.

#16 Updated by Nathan Cutler about 7 years ago

Assignee changed from Sandon Van Ness to Nathan Cutler

#17 Updated by Nathan Cutler about 7 years ago

Priority changed from Immediate to Normal

#18 Updated by J. Eric Ivancich about 7 years ago

Thank you, Nathan.

Nathan Cutler wrote: