Bug #46299
Updated by Sebastian Wagner almost 4 years ago
https://pulpito.ceph.com/swagner-2020-07-01_09:26:21-rados:cephadm-wip-swagner-testing-2020-07-01-0956-distro-basic-smithi/5194228/ <pre> 2020-07-01T10:04:28.253 INFO:tasks.cephadm:Adding local image mirror vossi04.front.sepia.ceph.com:5000 2020-07-01T10:04:28.301 DEBUG:teuthology.orchestra.remote:smithi189:/etc/containers/registries.conf is 4KB 2020-07-01T10:04:28.340 INFO:teuthology.orchestra.run.smithi189:> sudo sh -c 'cat > /etc/containers/registries.conf' 2020-07-01T10:04:28.400 DEBUG:teuthology.orchestra.remote:smithi205:/etc/containers/registries.conf is 4KB 2020-07-01T10:04:28.447 INFO:teuthology.orchestra.run.smithi205:> sudo sh -c 'cat > /etc/containers/registries.conf' ... 2020-07-01T10:14:26.759 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: debug 2020-07-01T10:14:26.700+0000 7f9405ffb700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:INFO:cephadm:Deploy daemon prometheus.a ... 2020-07-01T10:14:26.759 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:Verifying port 9095 ... 2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:Non-zero exit code 125 from /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=prom/prometheus:v2.18.1 -e NODE_NAME=smithi205 --entrypoint stat prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.access.redhat.com/prom/prometheus:v2.18.1... 2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr name unknown: Repo not found 2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.fedoraproject.org/prom/prometheus:v2.18.1... 2020-07-01T10:14:26.760 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr manifest unknown: manifest unknown 2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull registry.centos.org/prom/prometheus:v2.18.1... 2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr manifest unknown: manifest unknown 2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:10:18Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]" 2020-07-01T10:14:26.761 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:11:20Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]" 2020-07-01T10:14:26.762 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:12:22Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]" 2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr time="2020-07-01T10:13:24Z" level=error msg="HEADER map[Cache-Control:[no-cache] Content-Type:[application/json] Retry-After:[60]]" 2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr too many request to registry 2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr Error: unable to pull prom/prometheus:v2.18.1: 4 errors occurred: 2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.access.redhat.com/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.access.redhat.com/prom/prometheus: name unknown: Repo not found 2020-07-01T10:14:26.763 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.fedoraproject.org/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.fedoraproject.org/prom/prometheus: manifest unknown: manifest unknown 2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error initializing source docker://registry.centos.org/prom/prometheus:v2.18.1: Error reading manifest v2.18.1 in registry.centos.org/prom/prometheus: manifest unknown: manifest unknown 2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr * Error parsing image configuration: too many request to registry 2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: INFO:cephadm:stat:stderr 2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: Traceback (most recent call last): 2020-07-01T10:14:26.764 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 4847, in <module> 2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 1187, in _default_image 2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2886, in command_deploy 2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2818, in extract_uid_gid_monitoring 2020-07-01T10:14:26.765 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 1803, in extract_uid_gid 2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 2280, in run 2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: File "<stdin>", line 866, in call_throws 2020-07-01T10:14:26.766 INFO:journalctl@ceph.mgr.x.smithi205.stdout:Jul 01 10:14:26 smithi205 bash[31753]: RuntimeError: Failed command: /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=prom/prometheus:v2.18.1 -e NODE_NAME=smithi205 --entrypoint stat prom/prometheus:v2.18.1 -c %u %g /etc/prometheus </pre> Changing the registry mirror to `docker-mirror.front.sepia.ceph.com:5000` should work. I just can't do that myself.