Actions
Bug #46066
closed502 (Bad Gateway) when pulling from quay.ceph.io/ceph-ci/ceph
Status:
Resolved
Priority:
Normal
Assignee:
Category:
Infrastructure Service
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2020-06-18T07:04:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:Running command: /sbin/ip route ls
2020-06-18T07:04:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 102
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.183 metric 102
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:profile rt=0.0007503032684326172, stop=False, exit=0, reads=[4]
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:profile rt=0.0008108615875244141, stop=True, exit=0, reads=[4]
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Mon IP 172.21.15.183 is in CIDR network 172.21.0.0/20
2020-06-18T07:04:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Pulling latest quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9 container...
2020-06-18T07:04:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:Running command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
2020-06-18T07:04:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9...
2020-06-18T07:04:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=0.7787401676177979, stop=False, exit=None, reads=[6]
2020-06-18T07:04:33.768 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Getting image source signatures
2020-06-18T07:04:33.768 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=0.8956148624420166, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.008 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr invalid status code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.008 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1359565258026123, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.052 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9": una
ble to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9: unable to pull image: Error reading blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4: invalid st
atus code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.052 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1795868873596191, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1834368705749512, stop=False, exit=125, reads=[4, 6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1836388111114502, stop=True, exit=125, reads=[4, 6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9...
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Getting image source signatures
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr invalid status code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9": unab
le to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9: unable to pull image: Error reading blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4: invalid sta
tus code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:Traceback (most recent call last):
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr: File "/sbin/cephadm", line 4789, in <module>
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr: r = args.func()
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr: File "/sbin/cephadm", line 1170, in _default_image
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr: return func()
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr: File "/sbin/cephadm", line 2391, in command_bootstrap
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr: call_throws([container_path, 'pull', args.image])
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr: File "/sbin/cephadm", line 849, in call_throws
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr: raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-06-18T07:04:34.060 INFO:teuthology.orchestra.run.smithi183.stderr:RuntimeError: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
/a//kchai-2020-06-18_05:54:35-rados-wip-kefu-testing-2020-06-18-0846-distro-basic-smithi/5159557/
but the image was pushed to quay.ceph.io
+ docker push quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9 Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. Getting image source signatures Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd Copying blob sha256:1eb6ef00e50b2ea18c37524899a66b6e9badfbe0f13a6c99d1335b8ca9d07c33 Copying config sha256:313b89959e5d8ece93abf0eadd46deaf742f63fa557499423cb9d0074e970e8c Writing manifest to image destination Copying config sha256:313b89959e5d8ece93abf0eadd46deaf742f63fa557499423cb9d0074e970e8c Writing manifest to image destination Writing manifest to image destination Storing signatures
and it can be found at quay.ceph.io.
Updated by Sebastian Wagner almost 4 years ago
podman pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e Trying to pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e... Invalid status code returned when fetching blob 502 (Bad Gateway) Error: error pulling image "quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e": unable to pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e: unable to pull image: Error parsing image configuration: Invalid status code returned when fetching blob 502 (Bad Gateway)
Updated by Sebastian Wagner almost 4 years ago
now I'm getting
2020-06-18T12:10:45.550 INFO:teuthology.orchestra.run.smithi104.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22": unable to pull quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22: unable to pull image: Error initializing source docker://quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22: Error reading manifest 106d016f6707a24e1f4a79c35f52d9f7410fce22 in quay.ceph.io/ceph-ci/ceph: manifest unknown: manifest unknown
Updated by David Galloway almost 4 years ago
- Status changed from New to Resolved
The quay registry was using the Sepia Long Running Cluster for storage. Everything on the cluster except teuthology logs were inadvertently deleted around 0700 UTC.
quay service was restored around 1620 UTC.
Updated by Kefu Chai almost 4 years ago
- Status changed from Resolved to New
i am reopening this ticket:
2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr time="2020-06-19T04:54:30Z" level=error msg="Error pulling image ref //quay.ceph.io/ceph-ci/ceph:e3b9 02a109518526f1454b8880c78bec0564abbe: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid status code returned when fetching blob 502 (Bad Gateway)" 2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr Failed 2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe": unab le to pull quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe: unable to pull image: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid sta tus code returned when fetching blob 502 (Bad Gateway)
image built at https://shaman.ceph.com/builds/ceph/wip-kefu-testing-2020-06-19-1102/e3b902a109518526f1454b8880c78bec0564abbe/notcmalloc/212960/
+ docker push quay.ceph.io/ceph-ci/ceph:wip-kefu-testing-2020-06-19-1102-e3b902a-centos-8-x86_64-devel Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. Getting image source signatures Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4 Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Writing manifest to image destination Storing signatures + docker push quay.ceph.io/ceph-ci/ceph:wip-kefu-testing-2020-06-19-1102 Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. Getting image source signatures Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4 Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Writing manifest to image destination Storing signatures + docker push quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. Getting image source signatures Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4 Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e Writing manifest to image destination Writing manifest to image destination Storing signatures + return
the branch was pushed to ceph-ci after 2020-06-19-1102 GMT+8 (2020-06-19-0302 GMT), so i think it's after 2020-06-18-1620 GMT).
Updated by Sebastian Wagner almost 4 years ago
2020-06-19T11:26:03.998 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:23056ad0e6c4268c0acec7dcd44519ca2ab4d6a6...Getting image source signatures 2020-06-19T11:26:03.998 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:30b63e5b319bfcffb14c5167620e1e3f88b12c6b3260720cafb71814b86ef75b 2020-06-19T11:26:03.999 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr time="2020-06-19T11:26:03Z" level=error msg="Error pulling image ref //quay.ceph.io/ceph-ci/ceph:23056ad0e6c4268c0acec7dcd44519ca2ab4d6a6: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid status code returned when fetching blob 502 (Bad Gateway)"
as well
Updated by David Galloway almost 4 years ago
- Status changed from New to Resolved
I had to delete the repo and start over. I don't know enough about how container registries work to be able to explain it but from what I can tell, certain blobs relied on other blobs. Maybe they build off each other or something. But since a bunch of data was missing, I had to go nuclear.
I pushed container images based on Kefu's wip-kefu-testing-2020-06-19-1102 branch and was able to pull them to a smithi so now this issue is resolved.
Actions