Tools » teuthology

From https://pulpito.ceph.com/pdonnell-2020-08-19_23:49:18-kcephfs-wip-pdonnell-testing-20200819.214155-distro-basic-smithi/5360370/

Failure Reason:
Command failed on smithi063 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'

From the teuthology.log file:

2020-08-20T03:14:41.208 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] krb5-workstation-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-workstation-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.208 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] libkadm5-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/libkadm5-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.210 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] krb5-workstation-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-workstation-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.212 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] krb5-libs-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-libs-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.232 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] libkadm5-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/libkadm5-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.233 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] krb5-libs-1.17-9.el8.x86_64.rpm: Status code: 503 for https://download-cc-rdu01.fedoraproject.org/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-libs-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.348 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] krb5-workstation-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-workstation-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.351 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] libkadm5-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/libkadm5-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.352 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] krb5-workstation-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-workstation-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.354 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] krb5-libs-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-libs-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.354 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] libkadm5-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/libkadm5-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.354 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] krb5-libs-1.17-9.el8.x86_64.rpm: Status code: 404 for http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/os/Packages/krb5-libs-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.363 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] libkadm5-1.17-9.el8.x86_64.rpm: Status code: 404 for http://packages.oit.ncsu.edu/centos/8/BaseOS/x86_64/os/Packages/libkadm5-1.17-9.el8.x86_64.rpm
2020-08-20T03:14:41.363 INFO:teuthology.orchestra.run.smithi063.stdout:[MIRROR] krb5-workstation-1.17-9.el8.x86_64.rpm: Status code: 404 for http://packages.oit.ncsu.edu/centos/8/BaseOS/x86_64/os/Packages/krb5-workstation-1.17-9.el8.x86_64.rpm

When installing some packages, the mirror server failed with 503 code, which means the mirror server temporarily not available, we should retry it later. But the yum tool just skip it and retry other mirrors, which't didn't contain them.

For the cephfs suites, there maybe will fire the many test cases at the same time, and for each test case it may fire several nodes to install tens of packages at the same time. This may cause the mirrors overloaded.

We need one safe method to retry it.