Project

General

Profile

Bug #49742

Mirror was added, but still "toomanyrequests: You have reached your pull rate limit"

Added by Sebastian Wagner 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/sage-2021-03-04_03:32:45-upgrade:octopus-x-wip-sage-testing-2021-03-03-1623-distro-basic-smithi/5932032/

2021-03-04T09:37:02.573 INFO:teuthology.task.ansible.out:changed: [smithi187.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.151068", "end": "2021-03-04 09:37:02.533718", "rc": 0, "start": "2021-03-04 09:37:02.382
650", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

2021-03-04T09:37:02.595 INFO:teuthology.task.ansible.out:Thursday 04 March 2021  09:37:02 +0000 (0:00:00.029)       0:05:43.526 ******** 

2021-03-04T09:37:02.632 INFO:teuthology.task.ansible.out:
TASK [container-host : Restart docker service] *********************************

2021-03-04T09:37:02.634 INFO:teuthology.task.ansible.out:skipping: [smithi187.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"}

2021-03-04T09:37:02.906 INFO:teuthology.task.ansible.out:
TASK [container-host : Add local docker.io registry mirror] ********************

2021-03-04T09:37:02.908 INFO:teuthology.task.ansible.out:changed: [smithi190.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.154900", "end": "2021-03-04 09:37:02.881964", "rc": 0, "start": "2021-03-04 09:37:02.727
064", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

and then later:

2021-03-04T10:04:19.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ccebb34a-7cd0-11eb-9063-001a4aab830c -- ceph orch host add smithi190
2021-03-04T10:06:28.682 INFO:teuthology.orchestra.run.smithi190.stderr:Non-zero exit code 125 from /bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-octopus -e NODE_NAME=smithi190docker.io/ceph/daemon-base:latest-octopus -c %u %g /var/lib/ceph
2021-03-04T10:06:28.683 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr Trying to pull docker.io/ceph/daemon-base:latest-octopus...
2021-03-04T10:06:28.683 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr time="2021-03-04T10:04:50Z" level=warning msg="failed, retrying in 1s ... (1/3). Error: Error initializing source docker://ceph/daemon-base:latest-octopus: Error readingmanifest latest-octopus in docker.io/ceph/daemon-base: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2021-03-04T10:06:28.683 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr time="2021-03-04T10:05:22Z" level=warning msg="failed, retrying in 2s ... (2/3). Error: Error initializing source docker://ceph/daemon-base:latest-octopus: Error readingmanifest latest-octopus in docker.io/ceph/daemon-base: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2021-03-04T10:06:28.684 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr time="2021-03-04T10:05:54Z" level=warning msg="failed, retrying in 4s ... (3/3). Error: Error initializing source docker://ceph/daemon-base:latest-octopus: Error readingmanifest latest-octopus in docker.io/ceph/daemon-base: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2021-03-04T10:06:28.684 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr   toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2021-03-04T10:06:28.684 INFO:teuthology.orchestra.run.smithi190.stderr:stat: stderr Error: Error initializing source docker://ceph/daemon-base:latest-octopus: Error reading manifest latest-octopus in docker.io/ceph/daemon-base: toomanyrequests: You havereached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2021-03-04T10:06:28.685 INFO:teuthology.orchestra.run.smithi190.stderr:Traceback (most recent call last):
2021-03-04T10:06:28.685 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 6151, in <module>
2021-03-04T10:06:28.685 INFO:teuthology.orchestra.run.smithi190.stderr:    r = args.func()
2021-03-04T10:06:28.685 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1309, in _infer_fsid
2021-03-04T10:06:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:    return func()
2021-03-04T10:06:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1346, in _infer_config
2021-03-04T10:06:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:    return func()
2021-03-04T10:06:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1392, in _infer_image
2021-03-04T10:06:28.686 INFO:teuthology.orchestra.run.smithi190.stderr:    return func()
2021-03-04T10:06:28.687 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 3493, in command_shell
2021-03-04T10:06:28.687 INFO:teuthology.orchestra.run.smithi190.stderr:    make_log_dir(args.fsid)
2021-03-04T10:06:28.687 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 1490, in make_log_dir
2021-03-04T10:06:28.687 INFO:teuthology.orchestra.run.smithi190.stderr:    uid, gid = extract_uid_gid()
2021-03-04T10:06:28.688 INFO:teuthology.orchestra.run.smithi190.stderr:  File "/home/ubuntu/cephtest/cephadm", line 2097, in extract_uid_gid
2021-03-04T10:06:28.688 INFO:teuthology.orchestra.run.smithi190.stderr:    raise RuntimeError('uid/gid not found')
2021-03-04T10:06:28.688 INFO:teuthology.orchestra.run.smithi190.stderr:RuntimeError: uid/gid not found


Related issues

Related to Orchestrator - Bug #49633: podman: ERROR (catatonit:2): failed to exec pid1: No such file or directory New

History

#1 Updated by Sebastian Wagner 2 months ago

  • Related to Bug #49633: podman: ERROR (catatonit:2): failed to exec pid1: No such file or directory added

Also available in: Atom PDF