Project

General

Profile

Bug #57871

rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"

Added by Laura Flores over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/

2022-10-14T12:23:44.320 INFO:teuthology.orchestra.run.smithi120.stdout:Updating Subscription Management repositories.
2022-10-14T12:23:44.500 INFO:teuthology.orchestra.run.smithi120.stdout:52 files removed
2022-10-14T12:23:44.546 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw
2022-10-14T12:23:44.877 INFO:teuthology.orchestra.run.smithi120.stdout:Updating Subscription Management repositories.
2022-10-14T12:23:45.487 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64                        268 kB/s |  81 kB     00:00
2022-10-14T12:23:45.772 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages                             37 kB/s | 9.4 kB     00:00
2022-10-14T12:23:46.059 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages                            5.6 kB/s | 1.5 kB     00:00
2022-10-14T12:23:47.389 INFO:teuthology.orchestra.run.smithi120.stdout:Copr repo for python3-asyncssh owned by ceph    2.6 kB/s | 3.5 kB     00:01
2022-10-14T12:23:48.531 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux              12 MB/s |  13 MB     00:01
2022-10-14T12:23:51.331 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras                                      521 kB/s |  24 kB     00:00
2022-10-14T12:23:52.125 INFO:teuthology.orchestra.run.smithi120.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre  64 MB/s |  47 MB     00:00
2022-10-14T12:23:59.709 INFO:teuthology.orchestra.run.smithi120.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS   54 MB/s |  24 MB     00:00
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr:Error:
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr: Problem: conflicting requests
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr:  - nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64
2022-10-14T12:24:04.724 INFO:teuthology.orchestra.run.smithi120.stdout:(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
2022-10-14T12:24:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-10-14T12:24:04.804 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 31, in nested
    vars.append(enter())
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/__init__.py", line 218, in install
    install_packages(ctx, package_list, config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/__init__.py", line 83, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/rpm.py", line 278, in _update_package_list_and_install
    .format(install_cmd=install_cmd, cpack=cpack)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/rpm.py", line 155, in _retry_if_failures_are_recoverable
    return remote.run(args=args, stderr=stderr, stdout=stdout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 1: 'sudo yum -y install ceph-radosgw'

History

#1 Updated by Laura Flores over 1 year ago

Affected jobs report this happening on rhel 8.6.

2022-10-14T12:16:31.954 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Red Hat Enterprise Linux" 
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8.6 (Ootpa)" 
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID="rhel" 
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="fedora" 
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8.6" 
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8" 
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.redhat.com/" 
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 

#2 Updated by Brad Hubbard over 1 year ago

I reproduced the issue using the config from http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/7065475/ and checked the machine.

[root@smithi134 ~]# dnf search librabbitmq
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:54 ago on Mon 17 Oct 2022 02:39:36 AM UTC.
No matches found.

On a machine connected to RHSM we can work out what repo it should be in.

[root@7c7e0eeb8fa8 /]# dnf list librabbitmq
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                       5.6 MB/s |  52 MB     00:09    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                    5.1 MB/s |  46 MB     00:09    
Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                                                                                              1.8 MB/s | 810 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                                                                                           3.8 MB/s | 3.0 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                                                                                                    70 kB/s |  20 kB     00:00    
Available Packages
librabbitmq.i686                                                                                           0.9.0-3.el8                                                                                         rhel-8-for-x86_64-baseos-rpms
librabbitmq.x86_64                                                                                         0.9.0-3.el8                                                                                         rhel-8-for-x86_64-baseos-rpms

See how many packages should be in that repo.

[root@7c7e0eeb8fa8 /]# dnf repolist -v rhel-8-for-x86_64-baseos-rpms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 0:02:13 ago on Mon Oct 17 02:48:35 2022.
Repo-id            : rhel-8-for-x86_64-baseos-rpms
Repo-name          : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo-status        : enabled
Repo-revision      : 1665492373
Repo-updated       : Tue Oct 11 12:46:13 2022
Repo-pkgs          : 11473
Repo-available-pkgs: 11470
Repo-size          : 19 G
Repo-baseurl       : https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os
Repo-expire        : 86400 second(s) (last: Mon Oct 17 02:48:04 2022)
Repo-filename      : /etc/yum.repos.d/redhat.repo
Total packages: 11473

Check the same channel on the smithi where the job is failing.

# dnf repolist -v rhel-8-for-x86_64-baseos-rpms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                                                                                                                                       123 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                                                                                                                                    166 kB/s | 2.8 kB     00:00    
Repo-id            : rhel-8-for-x86_64-baseos-rpms
Repo-name          : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo-status        : enabled
Repo-revision      : 1665638114
Repo-updated       : Thu 13 Oct 2022 05:28:15 AM UTC
Repo-pkgs          : 4
Repo-available-pkgs: 4
Repo-size          : 341 k
Repo-baseurl       : https://satellite.front.sepia.ceph.com/pulp/repos/Ceph/Library/content/dist/rhel8/8.6/x86_64/baseos/os
Repo-expire        : 1 second(s) (last: Mon 17 Oct 2022 02:50:03 AM UTC)
Repo-filename      : /etc/yum.repos.d/redhat.repo
Total packages: 4

That channel only has 4 packages in it so there is a problem with the Satellite configuration or syncing of that channel.

#3 Updated by Laura Flores over 1 year ago

Thanks so much for checking into this, Brad! I tried working out some ways of debugging this on Friday, but it helps to see from your analysis how things are connected.

We know there was a problem with the RHEL satellite recently; perhaps this is fallout from that. I can check further on Monday.

#4 Updated by Laura Flores over 1 year ago

Christina shared some helpful documentation with me on debugging the satellite:
https://wiki.sepia.ceph.com/doku.php?id=services:satellite#errors_and_fixes

#5 Updated by Laura Flores over 1 year ago

We would need to ssh to satellite.front.sepia.ceph.com, check for any OOM issues with `dmesg -T`, resolve this, and then resync the BaseOS repository with:

for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done

I would do so, except I do not have the permissions to ssh to the satellite.

Also available in: Atom PDF