Bug #57871
rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"
0%
Description
http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/
2022-10-14T12:23:44.320 INFO:teuthology.orchestra.run.smithi120.stdout:Updating Subscription Management repositories.
2022-10-14T12:23:44.500 INFO:teuthology.orchestra.run.smithi120.stdout:52 files removed
2022-10-14T12:23:44.546 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw
2022-10-14T12:23:44.877 INFO:teuthology.orchestra.run.smithi120.stdout:Updating Subscription Management repositories.
2022-10-14T12:23:45.487 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 268 kB/s | 81 kB 00:00
2022-10-14T12:23:45.772 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00
2022-10-14T12:23:46.059 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 5.6 kB/s | 1.5 kB 00:00
2022-10-14T12:23:47.389 INFO:teuthology.orchestra.run.smithi120.stdout:Copr repo for python3-asyncssh owned by ceph 2.6 kB/s | 3.5 kB 00:01
2022-10-14T12:23:48.531 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 12 MB/s | 13 MB 00:01
2022-10-14T12:23:51.331 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 521 kB/s | 24 kB 00:00
2022-10-14T12:23:52.125 INFO:teuthology.orchestra.run.smithi120.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 64 MB/s | 47 MB 00:00
2022-10-14T12:23:59.709 INFO:teuthology.orchestra.run.smithi120.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 54 MB/s | 24 MB 00:00
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr:Error:
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr: Problem: conflicting requests
2022-10-14T12:24:04.723 INFO:teuthology.orchestra.run.smithi120.stderr: - nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64
2022-10-14T12:24:04.724 INFO:teuthology.orchestra.run.smithi120.stdout:(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
2022-10-14T12:24:04.802 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-10-14T12:24:04.804 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/contextutil.py", line 31, in nested
vars.append(enter())
File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
return next(self.gen)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/__init__.py", line 218, in install
install_packages(ctx, package_list, config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/__init__.py", line 83, in install_packages
ctx, remote, pkgs[system_type], config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/rpm.py", line 278, in _update_package_list_and_install
.format(install_cmd=install_cmd, cpack=cpack)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/task/install/rpm.py", line 155, in _retry_if_failures_are_recoverable
return remote.run(args=args, stderr=stderr, stdout=stdout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/remote.py", line 525, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 1: 'sudo yum -y install ceph-radosgw'
History
#1 Updated by Laura Flores over 1 year ago
Affected jobs report this happening on rhel 8.6.
2022-10-14T12:16:31.954 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Red Hat Enterprise Linux"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8.6 (Ootpa)"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID="rhel"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="fedora"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8.6"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.redhat.com/"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
#2 Updated by Brad Hubbard over 1 year ago
I reproduced the issue using the config from http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/7065475/ and checked the machine.
[root@smithi134 ~]# dnf search librabbitmq Updating Subscription Management repositories. Last metadata expiration check: 0:00:54 ago on Mon 17 Oct 2022 02:39:36 AM UTC. No matches found.
On a machine connected to RHSM we can work out what repo it should be in.
[root@7c7e0eeb8fa8 /]# dnf list librabbitmq Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 5.6 MB/s | 52 MB 00:09 Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 5.1 MB/s | 46 MB 00:09 Red Hat Universal Base Image 8 (RPMs) - BaseOS 1.8 MB/s | 810 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - AppStream 3.8 MB/s | 3.0 MB 00:00 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 70 kB/s | 20 kB 00:00 Available Packages librabbitmq.i686 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms librabbitmq.x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms
See how many packages should be in that repo.
[root@7c7e0eeb8fa8 /]# dnf repolist -v rhel-8-for-x86_64-baseos-rpms Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. DNF version: 4.7.0 cachedir: /var/cache/dnf Last metadata expiration check: 0:02:13 ago on Mon Oct 17 02:48:35 2022. Repo-id : rhel-8-for-x86_64-baseos-rpms Repo-name : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) Repo-status : enabled Repo-revision : 1665492373 Repo-updated : Tue Oct 11 12:46:13 2022 Repo-pkgs : 11473 Repo-available-pkgs: 11470 Repo-size : 19 G Repo-baseurl : https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os Repo-expire : 86400 second(s) (last: Mon Oct 17 02:48:04 2022) Repo-filename : /etc/yum.repos.d/redhat.repo Total packages: 11473
Check the same channel on the smithi where the job is failing.
# dnf repolist -v rhel-8-for-x86_64-baseos-rpms Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile Updating Subscription Management repositories. DNF version: 4.7.0 cachedir: /var/cache/dnf Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 123 kB/s | 2.4 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 166 kB/s | 2.8 kB 00:00 Repo-id : rhel-8-for-x86_64-baseos-rpms Repo-name : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) Repo-status : enabled Repo-revision : 1665638114 Repo-updated : Thu 13 Oct 2022 05:28:15 AM UTC Repo-pkgs : 4 Repo-available-pkgs: 4 Repo-size : 341 k Repo-baseurl : https://satellite.front.sepia.ceph.com/pulp/repos/Ceph/Library/content/dist/rhel8/8.6/x86_64/baseos/os Repo-expire : 1 second(s) (last: Mon 17 Oct 2022 02:50:03 AM UTC) Repo-filename : /etc/yum.repos.d/redhat.repo Total packages: 4
That channel only has 4 packages in it so there is a problem with the Satellite configuration or syncing of that channel.
#3 Updated by Laura Flores over 1 year ago
Thanks so much for checking into this, Brad! I tried working out some ways of debugging this on Friday, but it helps to see from your analysis how things are connected.
We know there was a problem with the RHEL satellite recently; perhaps this is fallout from that. I can check further on Monday.
#4 Updated by Laura Flores over 1 year ago
Christina shared some helpful documentation with me on debugging the satellite:
https://wiki.sepia.ceph.com/doku.php?id=services:satellite#errors_and_fixes
#5 Updated by Laura Flores over 1 year ago
We would need to ssh to satellite.front.sepia.ceph.com, check for any OOM issues with `dmesg -T`, resolve this, and then resync the BaseOS repository with:
for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done
I would do so, except I do not have the permissions to ssh to the satellite.