Project

General

Profile

Actions
Copy link

Bug #58734

open

Errors during downloading metadata for repository 'epel' for centos 8 and rhel

Added by Laura Flores about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Example run:
https://pulpito.ceph.com/yuriw-2023-02-13_22:01:57-fs-wip-yuri2-testing-2023-02-09-0842-quincy-distro-default-smithi/

2023-02-15T06:12:47.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel
2023-02-15T06:12:49.311 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories.
2023-02-15T06:12:49.312 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity
2023-02-15T06:12:51.046 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph    7.3 kB/s | 3.5 kB     00:00
2023-02-15T06:12:52.864 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux              73 kB/s | 129 kB     00:01
2023-02-15T06:12:52.864 INFO:teuthology.orchestra.run.smithi102.stderr:Errors during downloading metadata for repository 'epel':
2023-02-15T06:12:52.864 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz (IP: 131.252.208.20)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://ftp.linux.ncsu.edu/pub/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz (IP: 152.7.106.23)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 (IP: 131.252.208.20)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz (IP: 8.43.85.72)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz (IP: 131.252.208.20)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirror.pnl.gov/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz (IP: 192.101.102.2)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 (IP: 8.43.85.72)
2023-02-15T06:12:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:  - Curl error (7): Couldn't connect to server for http://mirror.oss.ou.edu/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz [Failed to connect to mirror.oss.ou.edu port 80: Connection refused]
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz (IP: 152.19.134.145)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirror.pnl.gov/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 (IP: 192.101.102.2)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 (IP: 152.19.134.145)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz (IP: 152.19.134.145)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirror.pnl.gov/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz (IP: 192.101.102.2)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://ftp.linux.ncsu.edu/pub/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz (IP: 152.7.106.23)
2023-02-15T06:12:52.866 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirror.pnl.gov/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz (IP: 192.101.102.2)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://ftp.linux.ncsu.edu/pub/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz (IP: 152.7.106.23)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz (IP: 131.252.208.20)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz (IP: 152.19.134.145)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Curl error (7): Couldn't connect to server for http://mirror.oss.ou.edu/epel/8/Everything/x86_64/repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz [Failed to connect to mirror.oss.ou.edu port 80: Connection refused]
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz (IP: 8.43.85.72)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz (IP: 8.43.85.72)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Status code: 404 for http://ftp.linux.ncsu.edu/pub/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 (IP: 152.7.106.23)
2023-02-15T06:12:52.867 INFO:teuthology.orchestra.run.smithi102.stderr:  - Curl error (7): Couldn't connect to server for http://mirror.oss.ou.edu/epel/8/Everything/x86_64/repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz [Failed to connect to mirror.oss.ou.edu port 80: Connection refused]
2023-02-15T06:12:52.868 INFO:teuthology.orchestra.run.smithi102.stderr:  - Curl error (7): Couldn't connect to server for http://mirror.oss.ou.edu/epel/8/Everything/x86_64/repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 [Failed to connect to mirror.oss.ou.edu port 80: Connection refused]
2023-02-15T06:12:52.902 INFO:teuthology.orchestra.run.smithi102.stderr:Error: Failed to download metadata for repo 'epel': Yum repo downloading error: Downloading error(s): repodata/b36203aecde5dfe1ddbbebd5d370aebd2678e63207c8230832875819f7cef916-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/7a1ac7e0bc8f7c2f7213c6370e54fff456d13df7b9ff311d8979ad88e8e14b63-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success; repodata/a08b84d61debfecebca5380df06cdaaee842f0a1f993bbd53c6898fb75d68aa5-updateinfo.xml.bz2 - Cannot download, all mirrors were already tried without success
2023-02-15T06:12:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-02-15T06:12:52.995 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
Actions
Copy link
 #2

Updated by Laura Flores about 1 year ago

Another variation, "Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh'":

2023-02-15T03:29:28.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel
2023-02-15T03:29:31.988 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-8 - AppStream                             35 MB/s |  28 MB     00:00
2023-02-15T03:29:38.130 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-8 - BaseOS                                42 MB/s |  27 MB     00:00
2023-02-15T03:29:42.737 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-8 - Extras                               156 kB/s |  18 kB     00:00
2023-02-15T03:29:43.130 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-8 - PowerTools                            22 MB/s | 5.7 MB     00:00
2023-02-15T03:35:04.650 INFO:teuthology.orchestra.run.smithi046.stdout:Copr repo for python3-asyncssh owned by ceph    0.0  B/s |   0  B     05:20
2023-02-15T03:35:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:Errors during downloading metadata for repository 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh':
2023-02-15T03:35:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (6): Couldn't resolve host name for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Could not resolve host: download.copr.fedorainfracloud.org]
2023-02-15T03:35:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (7): Couldn't connect to server for https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/epel-8-x86_64/repodata/repomd.xml [Failed to connect to download.copr.fedorainfracloud.org port 443: Connection timed out]
2023-02-15T03:35:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:Error: Failed to download metadata for repo 'copr:copr.fedorainfracloud.org:ceph:python3-asyncssh': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
2023-02-15T03:40:26.580 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux             0.0  B/s |   0  B     05:21
2023-02-15T03:40:26.581 INFO:teuthology.orchestra.run.smithi046.stderr:Errors during downloading metadata for repository 'epel':
2023-02-15T03:40:26.581 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (6): Couldn't resolve host name for http://mirror.pnl.gov/epel/8/Everything/x86_64/repodata/repomd.xml [Could not resolve host: mirror.pnl.gov]
2023-02-15T03:40:26.581 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (7): Couldn't connect to server for http://mirror.oss.ou.edu/epel/8/Everything/x86_64/repodata/repomd.xml [Failed to connect to mirror.oss.ou.edu port 80: Connection timed out]
2023-02-15T03:40:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (6): Couldn't resolve host name for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/repomd.xml [Could not resolve host: download-cc-rdu01.fedoraproject.org]
2023-02-15T03:40:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (7): Couldn't connect to server for http://ftp.linux.ncsu.edu/pub/epel/8/Everything/x86_64/repodata/repomd.xml [Failed to connect to ftp.linux.ncsu.edu port 80: Connection timed out]
2023-02-15T03:40:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (6): Couldn't resolve host name for http://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata/repomd.xml [Could not resolve host: mirrors.cat.pdx.edu]
2023-02-15T03:40:26.582 INFO:teuthology.orchestra.run.smithi046.stderr:  - Curl error (6): Couldn't resolve host name for http://download-ib01.fedoraproject.org/pub/epel/8/Everything/x86_64/repodata/repomd.xml [Could not resolve host: download-ib01.fedoraproject.org]
2023-02-15T03:40:26.626 INFO:teuthology.orchestra.run.smithi046.stderr:Error: Failed to download metadata for repo 'epel': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
2023-02-15T03:40:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-02-15T03:40:26.657 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
Actions
Copy link
 #3

Updated by Laura Flores about 1 year ago

The problem is that the test is trying to pull packages with outdated hashes from https://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata.
See how "caa55fa7819792332bb887639ff132cda312d33d60b93b8f8ea70a3460771244" doesn't exist as an option:
https://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/repodata

Index of /epel/8/Everything/x86_64/repodata/

../
033c015aa32a6baf42351f5891dd7313b6adca78a1d8ab3..> 15-Feb-2023 02:22             9934832
1d8e7844d730e5275ada5198f3bdff03ec68bab92984c72..> 15-Feb-2023 02:23             8612528
43f027f8f4e3b74289cca355d17c89a011544dd46d9f205..> 15-Feb-2023 02:22             1597821
a3097291734a3616f0e10ee86ad1a21970f06c5cd49df92..> 15-Feb-2023 02:22             2889876
a7c859494223f5e3bb682c8024bb2b232b553da6b206bdc..> 15-Feb-2023 02:23             1737264
adf18d52e4003334a72c39a5267180b5fe01cdd460ebdce..> 15-Feb-2023 02:22              131817
c111c3334d4161ec7327c779290f98b6a78299d35eb7fc7..> 15-Feb-2023 02:22               25524
d542f344acd4f6ed55637f992f687ba73a574c003a56a57..> 15-Feb-2023 02:23             4356624
da171eaab710a8091678fa9dd0548c13171fe530ca9f98a..> 15-Feb-2023 02:23                1884
fa7a1c92a6ec48aa9f77f5fef0bcbd8387e95a6860980cf..> 15-Feb-2023 02:37             1048326
repomd.xml                                         15-Feb-2023 02:37                4845

Looking at when repodata was last updated, we can see that it updated on 02/15/2023, which corresponds with the date of the failures:
https://mirrors.cat.pdx.edu/epel/8/Everything/x86_64/

Index of /epel/8/Everything/x86_64/

../
Packages/                                          15-Feb-2023 02:22                   -
debug/                                             19-Nov-2022 02:35                   -
drpms/                                             15-Feb-2023 02:23                   -
repodata/                                          15-Feb-2023 02:37                   -

We're testing a theory that we happened to hit the window between repo rebuild and mirrorlist update, which may explain the mismatched hashes.

Actions
Copy link
 #4

Updated by Laura Flores about 1 year ago

I tried reproducing the epel repository failure in a centos 8 steam container, but did not experience any 404s. See https://gist.github.com/ljflores/d4913e58b41c63dc1fd29e3a7e1e9444 for full output. This was done on a Fedora operating system:

$ podman pull centos:stream8
$ podman run -it centos:stream8
[root@a457bb81f6fd ~]# yum install -y epel-release
[root@a457bb81f6fd ~]# yum config-manager --set-enabled epel
[root@a457bb81f6fd ~]# yum install -y kernel
[root@a457bb81f6fd ~]# yum update

Actions
Copy link
 #5

Updated by Laura Flores about 1 year ago

Note from David Orman:
"It might be prudent to change the mirror list to be some subset we know/trust, and/or setup our own mirror in the future to avoid all of this"

Actions
Copy link
 #6

Updated by Laura Flores about 1 year ago

We are rerunning a previously-failed test run here:
https://pulpito.ceph.com/yuriw-2023-02-15_20:55:38-fs-wip-yuri2-testing-2023-02-09-0842-quincy-distro-default-smithi/

Already it looks like jobs are passing.

Actions
Copy link

Also available in: Atom PDF