Project

General

Profile

Bug #46066

502 (Bad Gateway) when pulling from quay.ceph.io/ceph-ci/ceph

Added by Kefu Chai about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Category:
Infrastructure Service
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature:

Description

2020-06-18T07:04:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:Running command: /sbin/ip route ls
2020-06-18T07:04:32.870 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 102
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.183 metric 102
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:profile rt=0.0007503032684326172, stop=False, exit=0, reads=[4]
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/sbin/ip:profile rt=0.0008108615875244141, stop=True, exit=0, reads=[4]
2020-06-18T07:04:32.871 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Mon IP 172.21.15.183 is in CIDR network 172.21.0.0/20
2020-06-18T07:04:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Pulling latest quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9 container...
2020-06-18T07:04:32.872 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:Running command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
2020-06-18T07:04:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9...
2020-06-18T07:04:33.651 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=0.7787401676177979, stop=False, exit=None, reads=[6]
2020-06-18T07:04:33.768 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Getting image source signatures
2020-06-18T07:04:33.768 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=0.8956148624420166, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.008 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr   invalid status code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.008 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1359565258026123, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.052 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9": una
ble to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9: unable to pull image: Error reading blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4: invalid st
atus code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.052 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1795868873596191, stop=False, exit=None, reads=[6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1834368705749512, stop=False, exit=125, reads=[4, 6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG:cephadm:/bin/podman:profile rt=1.1836388111114502, stop=True, exit=125, reads=[4, 6]
2020-06-18T07:04:34.056 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9...
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Getting image source signatures
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr   invalid status code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9": unab
le to pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9: unable to pull image: Error reading blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4: invalid sta
tus code from registry 502 (Bad Gateway)
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:Traceback (most recent call last):
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:  File "/sbin/cephadm", line 4789, in <module>
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:    r = args.func()
2020-06-18T07:04:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:  File "/sbin/cephadm", line 1170, in _default_image
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:    return func()
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:  File "/sbin/cephadm", line 2391, in command_bootstrap
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:    call_throws([container_path, 'pull', args.image])
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:  File "/sbin/cephadm", line 849, in call_throws
2020-06-18T07:04:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:    raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-06-18T07:04:34.060 INFO:teuthology.orchestra.run.smithi183.stderr:RuntimeError: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9

/a//kchai-2020-06-18_05:54:35-rados-wip-kefu-testing-2020-06-18-0846-distro-basic-smithi/5159557/

but the image was pushed to quay.ceph.io

+ docker push quay.ceph.io/ceph-ci/ceph:0a9c34cc8dbea56a321a6ea258a0652395f208f9
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Getting image source signatures
Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd
Copying blob sha256:1eb6ef00e50b2ea18c37524899a66b6e9badfbe0f13a6c99d1335b8ca9d07c33
Copying config sha256:313b89959e5d8ece93abf0eadd46deaf742f63fa557499423cb9d0074e970e8c
Writing manifest to image destination
Copying config sha256:313b89959e5d8ece93abf0eadd46deaf742f63fa557499423cb9d0074e970e8c
Writing manifest to image destination
Writing manifest to image destination
Storing signatures

see https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/41935//consoleFull

and it can be found at quay.ceph.io.

see https://quay.ceph.io/repository/ceph-ci/ceph/manifest/sha256:c87e08c732f1cfa73d3d9e7bd204645b897830beda036db487c2b5f4ebd088d1

History

#1 Updated by Sebastian Wagner about 2 months ago

podman pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e
Trying to pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e...
  Invalid status code returned when fetching blob 502 (Bad Gateway)
Error: error pulling image "quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e": unable to pull quay.ceph.io/ceph-ci/ceph:289329252c2ae943de01ac9f80a7a12c6964057e: unable to pull image: Error parsing image configuration: Invalid status code returned when fetching blob 502 (Bad Gateway)

#2 Updated by Sebastian Wagner about 2 months ago

now I'm getting

2020-06-18T12:10:45.550 INFO:teuthology.orchestra.run.smithi104.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22": unable to pull quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22: unable to pull image: Error initializing source docker://quay.ceph.io/ceph-ci/ceph:106d016f6707a24e1f4a79c35f52d9f7410fce22: Error reading manifest 106d016f6707a24e1f4a79c35f52d9f7410fce22 in quay.ceph.io/ceph-ci/ceph: manifest unknown: manifest unknown

http://pulpito.ceph.com/swagner-2020-06-18_11:45:20-rados:cephadm-wip-swagner-testing-2020-06-18-1155-distro-basic-smithi/

#3 Updated by David Galloway about 2 months ago

  • Status changed from New to Resolved

The quay registry was using the Sepia Long Running Cluster for storage. Everything on the cluster except teuthology logs were inadvertently deleted around 0700 UTC.

quay service was restored around 1620 UTC.

#4 Updated by Kefu Chai about 2 months ago

  • Status changed from Resolved to New

i am reopening this ticket:

2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr time="2020-06-19T04:54:30Z" level=error msg="Error pulling image ref //quay.ceph.io/ceph-ci/ceph:e3b9
02a109518526f1454b8880c78bec0564abbe: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid status code returned when fetching blob 502 (Bad Gateway)" 
2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr Failed
2020-06-19T04:54:30.494 INFO:teuthology.orchestra.run.smithi120.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe": unab
le to pull quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe: unable to pull image: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid sta
tus code returned when fetching blob 502 (Bad Gateway)

see http://pulpito.ceph.com/kchai-2020-06-19_04:33:35-rados-wip-kefu-testing-2020-06-19-1102-distro-basic-smithi/5162712/

image built at https://shaman.ceph.com/builds/ceph/wip-kefu-testing-2020-06-19-1102/e3b902a109518526f1454b8880c78bec0564abbe/notcmalloc/212960/

+ docker push quay.ceph.io/ceph-ci/ceph:wip-kefu-testing-2020-06-19-1102-e3b902a-centos-8-x86_64-devel
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Getting image source signatures
Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4
Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
+ docker push quay.ceph.io/ceph-ci/ceph:wip-kefu-testing-2020-06-19-1102
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Getting image source signatures
Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd
Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
+ docker push quay.ceph.io/ceph-ci/ceph:e3b902a109518526f1454b8880c78bec0564abbe
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Getting image source signatures
Copying blob sha256:aff9833d464191900b4a3b6e68aad912f930de639b411124e433aeb3757001cd
Copying blob sha256:17f7638240fd49f795451b6b42a3e967d3c4d2dd12d155bbfdda218b8c3725a4
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Copying config sha256:576fcca7162f45fbea0760adafe644bc216d08a220e4f7760ea2117a1aa54a2e
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
+ return

the branch was pushed to ceph-ci after 2020-06-19-1102 GMT+8 (2020-06-19-0302 GMT), so i think it's after 2020-06-18-1620 GMT).

see https://quay.ceph.io/repository/ceph-ci/ceph/manifest/sha256:3fef538ea1c9caad7c93ec70c95635f29b8e7706be5b1928a365c136d1b6ad9b

#5 Updated by Sebastian Wagner about 2 months ago

2020-06-19T11:26:03.998 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.ceph.io/ceph-ci/ceph:23056ad0e6c4268c0acec7dcd44519ca2ab4d6a6...Getting image source signatures
2020-06-19T11:26:03.998 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:30b63e5b319bfcffb14c5167620e1e3f88b12c6b3260720cafb71814b86ef75b
2020-06-19T11:26:03.999 INFO:teuthology.orchestra.run.smithi145.stderr:INFO:cephadm:/bin/podman:stderr time="2020-06-19T11:26:03Z" level=error msg="Error pulling image ref //quay.ceph.io/ceph-ci/ceph:23056ad0e6c4268c0acec7dcd44519ca2ab4d6a6: Error reading blob sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23: Invalid status code returned when fetching blob 502 (Bad Gateway)" 

as well

#6 Updated by David Galloway about 2 months ago

  • Status changed from New to Resolved

I had to delete the repo and start over. I don't know enough about how container registries work to be able to explain it but from what I can tell, certain blobs relied on other blobs. Maybe they build off each other or something. But since a bunch of data was missing, I had to go nuclear.

I pushed container images based on Kefu's wip-kefu-testing-2020-06-19-1102 branch and was able to pull them to a smithi so now this issue is resolved.

Also available in: Atom PDF