Project

General

Profile

Actions
Copy link

Bug #45343

closed

Error: error pulling image "quay.io/ceph-ci/ceph:" "net/http: TLS handshake timeout"

Added by Brad Hubbard almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
David Galloway
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2020-04-28_21:58:13-rados-wip-yuri-testing-2020-04-24-1941-master-distro-basic-smithi/4995287

2020-04-28T22:42:12.936 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:Mon IP 172.21.15.8 is in CIDR network 172.21.0.0/20
2020-04-28T22:42:12.937 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:Pulling latest quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716 container...
2020-04-28T22:42:12.937 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:Running command: /bin/podman pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716
2020-04-28T22:42:14.378 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716...
2020-04-28T22:42:14.923 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:/bin/podman:stderr Getting image source signatures
2020-04-28T22:42:15.106 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:/bin/podman:stderr Copying blob sha256:692477d6fc20e3db43464e7f3a7325291cad60d5295fb49671f30839ce8ed7e3
2020-04-28T22:42:25.034 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:/bin/podman:stderr   Get https://cdn.quay.io/sha256/8a/8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07?Expires=1588114334&Signature=SzDkwlzAjXlqCmbo~z~OK5nk3XIy6YVLZo3~XeeeDCwLRMWR4s~u6RT6Qbts-6dt8m1NuDizKrZh3kK6zEpVjP8nbzHQWKP5Q7FnyJi0a-84sLYZp-dAFiS38IYHC2SAXeidFLX1TQwS76BXU1vw3VQh~dhh~YVP5vnEsK01gPLCpIK-0ejJoOOOQfV-c4VPeZNTJyU4rAPkt1TgB~7bWcl3d~HJiQDP5VaGHiI5IYpEW8ZEJMmRuUlDr4uCBTzdeppAVgepjz-3Ck0veq2k6HYf1c1Ox5lv3PsDywlDMjjpXt4My-lg0A2AysU~NMqmDZzNa2VasphlS30Hhrjgxg__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-04-28T22:42:25.123 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716": unable to pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716: unable to pull image: Error reading blob sha256:8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07: Get https://cdn.quay.io/sha256/8a/8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07?Expires=1588114334&Signature=SzDkwlzAjXlqCmbo~z~OK5nk3XIy6YVLZo3~XeeeDCwLRMWR4s~u6RT6Qbts-6dt8m1NuDizKrZh3kK6zEpVjP8nbzHQWKP5Q7FnyJi0a-84sLYZp-dAFiS38IYHC2SAXeidFLX1TQwS76BXU1vw3VQh~dhh~YVP5vnEsK01gPLCpIK-0ejJoOOOQfV-c4VPeZNTJyU4rAPkt1TgB~7bWcl3d~HJiQDP5VaGHiI5IYpEW8ZEJMmRuUlDr4uCBTzdeppAVgepjz-3Ck0veq2k6HYf1c1Ox5lv3PsDywlDMjjpXt4My-lg0A2AysU~NMqmDZzNa2VasphlS30Hhrjgxg__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-04-28T22:42:25.128 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:Non-zero exit code 125 from /bin/podman pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716
2020-04-28T22:42:25.128 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716...
2020-04-28T22:42:25.128 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:/bin/podman:stderr Getting image source signatures
2020-04-28T22:42:25.129 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:692477d6fc20e3db43464e7f3a7325291cad60d5295fb49671f30839ce8ed7e3
2020-04-28T22:42:25.130 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:/bin/podman:stderr   Get https://cdn.quay.io/sha256/8a/8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07?Expires=1588114334&Signature=SzDkwlzAjXlqCmbo~z~OK5nk3XIy6YVLZo3~XeeeDCwLRMWR4s~u6RT6Qbts-6dt8m1NuDizKrZh3kK6zEpVjP8nbzHQWKP5Q7FnyJi0a-84sLYZp-dAFiS38IYHC2SAXeidFLX1TQwS76BXU1vw3VQh~dhh~YVP5vnEsK01gPLCpIK-0ejJoOOOQfV-c4VPeZNTJyU4rAPkt1TgB~7bWcl3d~HJiQDP5VaGHiI5IYpEW8ZEJMmRuUlDr4uCBTzdeppAVgepjz-3Ck0veq2k6HYf1c1Ox5lv3PsDywlDMjjpXt4My-lg0A2AysU~NMqmDZzNa2VasphlS30Hhrjgxg__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-04-28T22:42:25.130 INFO:teuthology.orchestra.run.smithi008.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716": unable to pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716: unable to pull image: Error reading blob sha256:8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07: Get https://cdn.quay.io/sha256/8a/8a29a15cefaeccf6545f7ecf11298f9672d2f0cdaf9e357a95133ac3ad3e1f07?Expires=1588114334&Signature=SzDkwlzAjXlqCmbo~z~OK5nk3XIy6YVLZo3~XeeeDCwLRMWR4s~u6RT6Qbts-6dt8m1NuDizKrZh3kK6zEpVjP8nbzHQWKP5Q7FnyJi0a-84sLYZp-dAFiS38IYHC2SAXeidFLX1TQwS76BXU1vw3VQh~dhh~YVP5vnEsK01gPLCpIK-0ejJoOOOQfV-c4VPeZNTJyU4rAPkt1TgB~7bWcl3d~HJiQDP5VaGHiI5IYpEW8ZEJMmRuUlDr4uCBTzdeppAVgepjz-3Ck0veq2k6HYf1c1Ox5lv3PsDywlDMjjpXt4My-lg0A2AysU~NMqmDZzNa2VasphlS30Hhrjgxg__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-04-28T22:42:25.131 INFO:teuthology.orchestra.run.smithi008.stderr:Traceback (most recent call last):
2020-04-28T22:42:25.131 INFO:teuthology.orchestra.run.smithi008.stderr:  File "/home/ubuntu/cephtest/cephadm", line 4585, in <module>
2020-04-28T22:42:25.132 INFO:teuthology.orchestra.run.smithi008.stderr:    r = args.func()
2020-04-28T22:42:25.132 INFO:teuthology.orchestra.run.smithi008.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1124, in _default_image
2020-04-28T22:42:25.132 INFO:teuthology.orchestra.run.smithi008.stderr:    return func()
2020-04-28T22:42:25.132 INFO:teuthology.orchestra.run.smithi008.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2294, in command_bootstrap
2020-04-28T22:42:25.133 INFO:teuthology.orchestra.run.smithi008.stderr:    call_throws([container_path, 'pull', args.image])
2020-04-28T22:42:25.133 INFO:teuthology.orchestra.run.smithi008.stderr:  File "/home/ubuntu/cephtest/cephadm", line 837, in call_throws
2020-04-28T22:42:25.134 INFO:teuthology.orchestra.run.smithi008.stderr:    raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-04-28T22:42:25.134 INFO:teuthology.orchestra.run.smithi008.stderr:RuntimeError: Failed command: /bin/podman pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716
2020-04-28T22:42:25.135 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:Releasing lock 140049454935568 on /run/cephadm/7fcd4600-89a1-11ea-a068-001a4aab830c.lock
2020-04-28T22:42:25.135 INFO:teuthology.orchestra.run.smithi008.stderr:DEBUG:cephadm:Lock 140049454935568 released on /run/cephadm/7fcd4600-89a1-11ea-a068-001a4aab830c.lock
2020-04-28T22:42:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1

$ podman pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716
Trying to pull quay.io/ceph-ci/ceph:1674bb8f59f2c0d240ee17d0ef66d5eafc97f716...
Getting image source signatures
Copying blob 8a29a15cefae skipped: already exists  
Copying blob 692477d6fc20 done  
Copying config 95e5ea2808 done  
Writing manifest to image destination
Storing signatures
95e5ea2808d268fe814803deb7e090446b7b98e936744ef242e30b89fa9d81b4

This is probably a transient failure since I was able to pull the image with no problem but creating this tracker in case it shows up more regularly.

Related issues 1 (0 open — 1 closed)

Related to Orchestrator - Bug #46271: podman pull: transient "Error: error creating container storage: error creating read-write layer with ID" failureResolvedSebastian Wagner

Actions
Actions
Copy link
 #1

Updated by Brad Hubbard almost 4 years ago

/a/yuriw-2020-05-22_19:55:53-rados-wip-yuri-master_5.22.20-distro-basic-smithi/5083366

2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c -e NODE_NAME=smithi192 --entrypoint stat quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c -c %u %g /var/lib/ceph
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Trying to pull quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c...
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Getting image source signatures
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.741 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.742 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.743 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-23T00:08:39.743 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Copying blob sha256:7101dc8adceb21bd3231d98204903a87d99ae08d692a9c414dec3df4034b2657
2020-05-23T00:08:39.743 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr   Get https://cdn.quay.io/sha256/d7/d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4?Expires=1590193108&Signature=X-ZYjS-Be6KTE2RErPYGtoXYsIlyQuQYE44mg6OVlfebK4zzQoUYFAsbubrh-jVkck3fzMyVKSbolAP36Gev1jTRcXPBVJqbbe4ISzVJoLPuvgm~N3gExx9x9RE7axB2I9-y8FSKI3fBVnSeuNvhzs-MZuIcEwftpljOTBqenqS9v6AXF9i2EeZ6pNyEymyCKxpA9RKmCaVIHv597iT~10F4tTbAWnuO-BRljE3KhVuVjl5YsFLBdyKDZW94nNT-yL-GoVw3LZf5G-BzJngaZgzOZ1iIzSCr0h5YUBQwSQMmutWi~LoRv~qFjph1a317pLa-wW0qg0xZDT6LKGsXmQ__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-05-23T00:08:39.743 INFO:teuthology.orchestra.run.smithi192.stderr:INFO:cephadm:stat:stderr Error: unable to pull quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c: unable to pull image: Error reading blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4: Get https://cdn.quay.io/sha256/d7/d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4?Expires=1590193108&Signature=X-ZYjS-Be6KTE2RErPYGtoXYsIlyQuQYE44mg6OVlfebK4zzQoUYFAsbubrh-jVkck3fzMyVKSbolAP36Gev1jTRcXPBVJqbbe4ISzVJoLPuvgm~N3gExx9x9RE7axB2I9-y8FSKI3fBVnSeuNvhzs-MZuIcEwftpljOTBqenqS9v6AXF9i2EeZ6pNyEymyCKxpA9RKmCaVIHv597iT~10F4tTbAWnuO-BRljE3KhVuVjl5YsFLBdyKDZW94nNT-yL-GoVw3LZf5G-BzJngaZgzOZ1iIzSCr0h5YUBQwSQMmutWi~LoRv~qFjph1a317pLa-wW0qg0xZDT6LKGsXmQ__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:Traceback (most recent call last):
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 4655, in <module>
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:    r = args.func()
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1059, in _infer_fsid
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:    return func()
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1090, in _infer_config
2020-05-23T00:08:39.744 INFO:teuthology.orchestra.run.smithi192.stderr:    return func()
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1136, in _infer_image
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:    return func()
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2776, in command_shell
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:    make_log_dir(args.fsid)
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1220, in make_log_dir
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:    (uid, gid) = extract_uid_gid()
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1734, in extract_uid_gid
2020-05-23T00:08:39.745 INFO:teuthology.orchestra.run.smithi192.stderr:    args=['-c', '%u %g', file_path]
2020-05-23T00:08:39.746 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2183, in run
2020-05-23T00:08:39.754 INFO:teuthology.orchestra.run.smithi192.stderr:    self.run_cmd(), desc=self.entrypoint, timeout=timeout)
2020-05-23T00:08:39.754 INFO:teuthology.orchestra.run.smithi192.stderr:  File "/home/ubuntu/cephtest/cephadm", line 837, in call_throws
2020-05-23T00:08:39.755 INFO:teuthology.orchestra.run.smithi192.stderr:    raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-05-23T00:08:39.755 INFO:teuthology.orchestra.run.smithi192.stderr:RuntimeError: Failed command: /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c -e NODE_NAME=smithi192 --entrypoint stat quay.io/ceph-ci/ceph:c3321b7686f181e1bcb805a1fb24baced390ae4c -c %u %g /var/lib/ceph
2020-05-23T00:08:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1
Actions
Copy link
 #2

Updated by Brad Hubbard almost 4 years ago

  • Priority changed from Low to Normal
  • Severity changed from 4 - irritation to 3 - minor
Actions
Copy link
 #3

Updated by Sebastian Wagner almost 4 years ago

  • Assignee set to David Galloway

David, do you have an idea how to make quay.io reliable?

Actions
Copy link
 #4

Updated by David Galloway almost 4 years ago

Sebastian Wagner wrote:

David, do you have an idea how to make quay.io reliable?

I'm still a container noob. Seems this would also be resolved by having our own internal registry, no?

Actions
Copy link
 #5

Updated by Kefu Chai almost 4 years ago 

2020-05-28T01:13:40.426 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:41868f1d9421d7d6825c37a87e1845b67ef9255e...
2020-05-28T01:13:40.426 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Getting image source signatures
2020-05-28T01:13:40.426 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:d71c5eab96b2ab4ce9461534cf5e4beda459d7dd3049e9ee2a4c73a13c0629b4
2020-05-28T01:13:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.427 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.428 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.429 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.429 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
2020-05-28T01:13:40.429 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Copying blob sha256:1bf02769ae5995f54ef4294f73e9bd67ab8d90f410c91daf703b47c774d2153f
2020-05-28T01:13:40.429 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr   Get https://cdn.quay.io/sharedimages/86f0a285-6f29-47c4-a3ae-7e2c70cad0ba/layer?Expires=1590629010&
Signature=eBhyy5sxEFc0NSw1QqyVajHIzxNvHB-I1kOqNBCjooXYqDEpRIWRidkg4KOpUB0IK2l4Bwg30u0WT4SOVlseuBZkAG0dhJz91JtqjP1Q0YeZY6bmO7WNaRBHUlENBryDe5b-xePnQEYpk2-bn8B8ViztCm3AsCMrAQ8TMTPpqtX-qFnNI8ZCsAW1R1oNDUJy7J
raDVHOueuWFWnXekmEbNg7lwL7ovcRyqgVmFzwGC3ZY0Y9T1aWZvQsvcm292mYSMCgnWiYewyOd4Nx~GmvSE5Pq3iolu1OfnG4FB3n6otIFkcA6weXrG6SCmu7htK2fVg27bKj1k2tOjI-XgdPtA__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS hands
hake timeout
2020-05-28T01:13:40.429 INFO:teuthology.orchestra.run.smithi169.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:41868f1d9421d7d6825c37a87e1845b67ef9255e": unable to
 pull quay.io/ceph-ci/ceph:41868f1d9421d7d6825c37a87e1845b67ef9255e: unable to pull image: Error reading blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4: Get https://cdn.quay.
io/sharedimages/86f0a285-6f29-47c4-a3ae-7e2c70cad0ba/layer?Expires=1590629010&Signature=eBhyy5sxEFc0NSw1QqyVajHIzxNvHB-I1kOqNBCjooXYqDEpRIWRidkg4KOpUB0IK2l4Bwg30u0WT4SOVlseuBZkAG0dhJz91JtqjP1Q0YeZY6bmO7WN
aRBHUlENBryDe5b-xePnQEYpk2-bn8B8ViztCm3AsCMrAQ8TMTPpqtX-qFnNI8ZCsAW1R1oNDUJy7JraDVHOueuWFWnXekmEbNg7lwL7ovcRyqgVmFzwGC3ZY0Y9T1aWZvQsvcm292mYSMCgnWiYewyOd4Nx~GmvSE5Pq3iolu1OfnG4FB3n6otIFkcA6weXrG6SCmu7htK2
fVg27bKj1k2tOjI-XgdPtA__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA: net/http: TLS handshake timeout
2020-05-28T01:13:40.430 INFO:teuthology.orchestra.run.smithi169.stderr:Traceback (most recent call last):
2020-05-28T01:13:40.430 INFO:teuthology.orchestra.run.smithi169.stderr:  File "/home/ubuntu/cephtest/cephadm", line 4655, in <module>
2020-05-28T01:13:40.430 INFO:teuthology.orchestra.run.smithi169.stderr:    r = args.func()
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1153, in _default_image
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:    return func()
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2333, in command_bootstrap
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:    call_throws([container_path, 'pull', args.image])
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:  File "/home/ubuntu/cephtest/cephadm", line 837, in call_throws
2020-05-28T01:13:40.431 INFO:teuthology.orchestra.run.smithi169.stderr:    raise RuntimeError('Failed command: %s' % ' '.join(command))
2020-05-28T01:13:40.432 INFO:teuthology.orchestra.run.smithi169.stderr:RuntimeError: Failed command: /bin/podman pull quay.io/ceph-ci/ceph:41868f1d9421d7d6825c37a87e1845b67ef9255e
2020-05-28T01:13:40.432 INFO:teuthology.orchestra.run.smithi169.stderr:DEBUG:cephadm:Releasing lock 139688472225216 on /run/cephadm/6fd76b68-a080-11ea-a06a-001a4aab830c.lock
2020-05-28T01:13:40.432 INFO:teuthology.orchestra.run.smithi169.stderr:DEBUG:cephadm:Lock 139688472225216 released on /run/cephadm/6fd76b68-a080-11ea-a06a-001a4aab830c.lock

/a/kchai-2020-05-27_23:43:53-rados-wip-kefu-testing-2020-05-27-2242-distro-basic-smithi/5097119

Actions
Copy link
 #6

Updated by Sebastian Wagner almost 4 years ago

David Galloway wrote:

Sebastian Wagner wrote:

David, do you have an idea how to make quay.io reliable?

I'm still a container noob. Seems this would also be resolved by having our own internal registry, no?

Just created an upstream issue: https://issues.redhat.com/browse/PROJQUAY-745 . In case we're not getting a good answer, we could indeed add another mirror registry for quay.io, but that would be a different ballpark due to the amount of images we create.

Actions
Copy link
 #7

Updated by Sebastian Wagner almost 4 years ago

Not getting any better:

2020-05-28T15:41:50.519 INFO:teuthology.orchestra.run.smithi109.stderr:DEBUG:cephadm:Running command: /bin/podman pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59
2020-05-28T15:41:51.916 INFO:teuthology.orchestra.run.smithi109.stderr:DEBUG:cephadm:/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59...
2020-05-28T15:41:52.048 INFO:teuthology.orchestra.run.smithi109.stderr:DEBUG:cephadm:/bin/podman:stderr   unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aceph-ci%2Fceph%3Apull&service=quay.io
2020-05-28T15:41:52.125 INFO:teuthology.orchestra.run.smithi109.stderr:DEBUG:cephadm:/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59": unable to pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59: unable to pull image: Error initializing source docker://quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aceph-ci%2Fceph%3Apull&service=quay.io
2020-05-28T15:41:52.130 INFO:teuthology.orchestra.run.smithi109.stderr:INFO:cephadm:Non-zero exit code 125 from /bin/podman pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59
2020-05-28T15:41:52.130 INFO:teuthology.orchestra.run.smithi109.stderr:INFO:cephadm:/bin/podman:stderr Trying to pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59...
2020-05-28T15:41:52.130 INFO:teuthology.orchestra.run.smithi109.stderr:INFO:cephadm:/bin/podman:stderr   unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aceph-ci%2Fceph%3Apull&service=quay.io
2020-05-28T15:41:52.130 INFO:teuthology.orchestra.run.smithi109.stderr:INFO:cephadm:/bin/podman:stderr Error: error pulling image "quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59": unable to pull quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59: unable to pull image: Error initializing source docker://quay.io/ceph-ci/ceph:21b80bd2069c072591f32ad17da1c61574e6fc59: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?scope=repository%3Aceph-ci%2Fceph%3Apull&service=quay.io

http://qa-proxy.ceph.com/teuthology/swagner-2020-05-28_15:29:03-rados:cephadm-wip-swagner-testing-2020-05-27-1352-distro-basic-smithi/5098905/teuthology.log

Actions
Copy link
 #8

Updated by David Galloway almost 4 years ago

Okay, I set up a docker mirror and a quay registry today.

docker-mirror.front.sepia.ceph.com

$ podman pull --tls-verify=false docker-mirror.front.sepia.ceph.com:5000/grafana/grafana
Trying to pull docker-mirror.front.sepia.ceph.com:5000/grafana/grafana...
Getting image source signatures
Copying blob 55d80fcaa5e7 done  
Copying blob f04b241748a1 done  
Copying blob 83ed1c0a8b40 done  
Copying blob cbdbe7a5bc2a done  
Copying blob 47d3315d65dd done  
Copying blob d7be2e229d83 done  
Copying config 465c4af545 done  
Writing manifest to image destination
Storing signatures
465c4af545b156014b9a7ca526d5e3577f1b0d5c4a37344ea1d25b0bed89bce4

I can't figure out the appropriate /etc/containers/registries.conf syntax.

quay.ceph.io

This is just a matter of s/quay.io/quay.ceph.io in ceph-build.git. I created the same user (Dan's) and repo: https://quay.ceph.io/repository/ceph-ci/ceph?tab=tags

teuthology or ceph.git (or wherever that gets defined) needs to be updated too.

Actions
Copy link
 #9

Updated by Sebastian Wagner almost 4 years ago 

DEBU[0000] Ping https://docker-mirror.front.sepia.ceph.com:5000/v2/ err Get https://docker-mirror.front.sepia.ceph.com:5000/v2/: http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://docker-mirror.front.sepia.ceph.com:5000/v2/", Err:(*errors.errorString)(0xc420440070)})

using

unqualified-search-registries = ["docker.io"]

[[registry]]
location = "docker.io" 

[[registry.mirror]]
#location = "vossi04.front.sepia.ceph.com:5000" 
location = "docker-mirror.front.sepia.ceph.com:5000" 
insecure = true
Actions
Copy link
 #10

Updated by David Galloway almost 4 years ago

Yeah, that's what I meant when I said I can't figure out the appropriate /etc/containers/registries.conf syntax. :)

podman pull --tls-verify=false docker-mirror.front.sepia.ceph.com:5000/grafana/grafana works. If someone can figure out the registries.conf syntax, even better.
Actions
Copy link
 #11

Updated by Sebastian Wagner almost 4 years ago 

#
# Usage:
#  podman run -d -p 5000:5000 --name registry -v `pwd`/config.yml:/etc/docker/registry/config.yml -v "$(pwd)"/certs:/certs  --security-opt label=disable registry:2
# 
# Plus something like 
# unqualified-search-registries = ["docker.io"]
# [[registry]]
# location = "docker.io" 
# 
# [[registry.mirror]]
# location = "vossi04.front.sepia.ceph.com:5000" 
# insecure = true

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://registry-1.docker.io
Actions
Copy link
 #12

Updated by Sebastian Wagner almost 4 years ago

quay.io closed the ticket with "it's your job to do retries"

Actions
Copy link
 #13

Updated by David Galloway almost 4 years ago

  • Status changed from New to In Progress

Sebastian Wagner wrote:

quay.io closed the ticket with "it's your job to do retries"

lol. Well, quay.ceph.io is available. I'm happy to switch the CI over to pushing dev container builds there. I don't know what's involved on the teuthology end of things though.

Actions
Copy link
 #14

Updated by David Galloway almost 4 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link
 #15

Updated by Sebastian Wagner almost 4 years ago

  • Related to Bug #46271: podman pull: transient "Error: error creating container storage: error creating read-write layer with ID" failure added
Actions
Copy link

Also available in: Atom PDF