Bug #45618
closedcephadm tests fail because missing image on quay.io
0%
Description
2020-05-20T01:05:56.697 INFO:teuthology.orchestra.run.smithi060.stderr:DEBUG:cephadm:/usr/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314": unable to pull quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314: unable to pull image: Error initializing source docker://quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314: Error reading manifest dbd4f5a569cf99ab48a249804c55d37916a5a314 in quay.io/ceph-ci/ceph: manifest unknown: manifest unknown
2020-05-20T01:05:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:INFO:cephadm:Non-zero exit code 125 from /usr/bin/podman pull quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314
2020-05-20T01:05:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:INFO:cephadm:/usr/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314...
2020-05-20T01:05:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:INFO:cephadm:/usr/bin/podman:stderr manifest unknown: manifest unknown
2020-05-20T01:05:56.709 INFO:teuthology.orchestra.run.smithi060.stderr:INFO:cephadm:/usr/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314": unable to pull quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314: unable to pull image: Error initializing source docker://quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314: Error reading manifest dbd4f5a569cf99ab48a249804c55d37916a5a314 in quay.io/ceph-ci/ceph: manifest unknown: manifest unknown
2020-05-20T01:05:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:Traceback (most recent call last):
2020-05-20T01:05:56.714 INFO:teuthology.orchestra.run.smithi060.stderr: File "/home/ubuntu/cephtest/cephadm", line 4642, in <module>
2020-05-20T01:05:56.714 INFO:teuthology.orchestra.run.smithi060.stderr: r = args.func()
2020-05-20T01:05:56.714 INFO:teuthology.orchestra.run.smithi060.stderr: File "/home/ubuntu/cephtest/cephadm", line 1153, in _default_image
2020-05-20T01:05:56.714 INFO:teuthology.orchestra.run.smithi060.stderr: return func()
2020-05-20T01:05:56.715 INFO:teuthology.orchestra.run.smithi060.stderr: File "/home/ubuntu/cephtest/cephadm", line 2333, in command_bootstrap
2020-05-20T01:05:56.715 INFO:teuthology.orchestra.run.smithi060.stderr: call_throws([container_path, 'pull', args.image])
2020-05-20T01:05:56.715 INFO:teuthology.orchestra.run.smithi060.stderr: File "/home/ubuntu/cephtest/cephadm", line 837, in call_throws
2020-05-20T01:05:56.715 INFO:teuthology.orchestra.run.smithi060.stderr: raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-05-20T01:05:56.715 INFO:teuthology.orchestra.run.smithi060.stderr:RuntimeError: Failed command: /usr/bin/podman pull quay.io/ceph-ci/ceph:dbd4f5a569cf99ab48a249804c55d37916a5a314
seems we failed to push the image to quay.io
+ docker login -u **** -p **** quay.io Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. WARNING! Using --password via the cli is insecure. Please consider using --password-stdin Error: error authenticating creds for "quay.io": Requesting bear token: invalid status code from registry 504 (Gateway Timeout) + /bin/true + cd /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic + sudo rm -rf ceph-container + update_build_status completed ceph centos 8 x86_64
Updated by Sebastian Wagner almost 4 years ago
yesterday, quay.io returned "Bad Gateway" in my runs. I think this was an infrastructure issue at quay.io. I just scheduled new shaman builds. let's see if things work again.
Updated by Kefu Chai almost 4 years ago
- Status changed from New to Can't reproduce
May 19, 2020
Quay.io outage
Resolved - Currently service is restored and stable. At this time, the incident is resolved.
May 19, 22:40 EDT
Monitoring - We have successfully reverted service impacting changes. The service is currently operational and stable. We will continue to monitor. Next update in 30 minutes or as new information becomes available.
May 19, 22:11 EDT
Update - We are still investigating the root cause and continue to work with our infrastructure provider on restoring service. Next update in 30 minutes or as new information becomes available.
May 19, 21:22 EDT
...
May 18, 2020
Quay.io is seeing elevated error rates
Resolved - Service has returned to operational status.
May 18, 14:09 EDT
Monitoring - A fix has been implemented and we are monitoring the results.
May 18, 14:03 EDT
Investigating - We are currently seeing elevated error rate and are working on resolution.
May 18, 13:55 EDT
it's very likely caused by the outrage of quay.io