https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2022-10-14T22:19:04ZCeph Infrastructure - Bug #57871: rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"https://tracker.ceph.com/issues/57871?journal_id=2274632022-10-14T22:19:04ZLaura Flores
<ul></ul><p>Affected jobs report this happening on rhel 8.6.<br /><pre><code class="text syntaxhl"><span class="CodeRay">2022-10-14T12:16:31.954 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="Red Hat Enterprise Linux"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8.6 (Ootpa)"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID="rhel"
2022-10-14T12:16:32.014 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="fedora"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8.6"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-14T12:16:32.015 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://www.redhat.com/"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-14T12:16:32.016 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-14T12:16:32.017 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
</span></code></pre></p> Infrastructure - Bug #57871: rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"https://tracker.ceph.com/issues/57871?journal_id=2274722022-10-17T03:02:40ZBrad Hubbardbhubbard@redhat.com
<ul></ul><p>I reproduced the issue using the config from <a class="external" href="http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/7065475/">http://pulpito.front.sepia.ceph.com/yuriw-2022-10-13_17:24:48-rados-main-distro-default-smithi/7065475/</a> and checked the machine.</p>
<pre>
[root@smithi134 ~]# dnf search librabbitmq
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:54 ago on Mon 17 Oct 2022 02:39:36 AM UTC.
No matches found.
</pre>
<p>On a machine connected to RHSM we can work out what repo it should be in.</p>
<pre>
[root@7c7e0eeb8fa8 /]# dnf list librabbitmq
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 5.6 MB/s | 52 MB 00:09
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 5.1 MB/s | 46 MB 00:09
Red Hat Universal Base Image 8 (RPMs) - BaseOS 1.8 MB/s | 810 kB 00:00
Red Hat Universal Base Image 8 (RPMs) - AppStream 3.8 MB/s | 3.0 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 70 kB/s | 20 kB 00:00
Available Packages
librabbitmq.i686 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms
librabbitmq.x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms
</pre>
<p>See how many packages should be in that repo.</p>
<pre>
[root@7c7e0eeb8fa8 /]# dnf repolist -v rhel-8-for-x86_64-baseos-rpms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Last metadata expiration check: 0:02:13 ago on Mon Oct 17 02:48:35 2022.
Repo-id : rhel-8-for-x86_64-baseos-rpms
Repo-name : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo-status : enabled
Repo-revision : 1665492373
Repo-updated : Tue Oct 11 12:46:13 2022
Repo-pkgs : 11473
Repo-available-pkgs: 11470
Repo-size : 19 G
Repo-baseurl : https://cdn.redhat.com/content/dist/rhel8/8/x86_64/baseos/os
Repo-expire : 86400 second(s) (last: Mon Oct 17 02:48:04 2022)
Repo-filename : /etc/yum.repos.d/redhat.repo
Total packages: 11473
</pre>
<p>Check the same channel on the smithi where the job is failing.</p>
<pre>
# dnf repolist -v rhel-8-for-x86_64-baseos-rpms
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 123 kB/s | 2.4 kB 00:00
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 166 kB/s | 2.8 kB 00:00
Repo-id : rhel-8-for-x86_64-baseos-rpms
Repo-name : Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
Repo-status : enabled
Repo-revision : 1665638114
Repo-updated : Thu 13 Oct 2022 05:28:15 AM UTC
Repo-pkgs : 4
Repo-available-pkgs: 4
Repo-size : 341 k
Repo-baseurl : https://satellite.front.sepia.ceph.com/pulp/repos/Ceph/Library/content/dist/rhel8/8.6/x86_64/baseos/os
Repo-expire : 1 second(s) (last: Mon 17 Oct 2022 02:50:03 AM UTC)
Repo-filename : /etc/yum.repos.d/redhat.repo
Total packages: 4
</pre>
<p>That channel only has 4 packages in it so there is a problem with the Satellite configuration or syncing of that channel.</p> Infrastructure - Bug #57871: rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"https://tracker.ceph.com/issues/57871?journal_id=2274762022-10-17T04:47:18ZLaura Flores
<ul></ul><p>Thanks so much for checking into this, Brad! I tried working out some ways of debugging this on Friday, but it helps to see from your analysis how things are connected.</p>
<p>We know there was a problem with the RHEL satellite recently; perhaps this is fallout from that. I can check further on Monday.</p> Infrastructure - Bug #57871: rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"https://tracker.ceph.com/issues/57871?journal_id=2275152022-10-17T20:14:15ZLaura Flores
<ul></ul><p>Christina shared some helpful documentation with me on debugging the satellite:<br /><a class="external" href="https://wiki.sepia.ceph.com/doku.php?id=services:satellite#errors_and_fixes">https://wiki.sepia.ceph.com/doku.php?id=services:satellite#errors_and_fixes</a></p> Infrastructure - Bug #57871: rhel 8 jobs failing from "nothing provides librabbitmq.so.4()(64bit) needed by ceph-radosgw-2:18.0.0-285.g6fb85925.el8.x86_64"https://tracker.ceph.com/issues/57871?journal_id=2275162022-10-17T21:59:05ZLaura Flores
<ul></ul><p>We would need to ssh to satellite.front.sepia.ceph.com, check for any OOM issues with `dmesg -T`, resolve this, and then resync the BaseOS repository with:</p>
<pre><code class="text syntaxhl"><span class="CodeRay">for id in $(hammer -c /etc/hammer/cli.modules.d/foreman.yml repository list | grep -i "baseos" | awk '{ print $1 }'); do hammer -c /etc/hammer/cli.modules.d/foreman.yml repository synchronize --async --validate-contents true --id $id; done
</span></code></pre>
<p>I would do so, except I do not have the permissions to ssh to the satellite.</p>