Ceph » Orchestrator

/a/lflores-2023-02-07_03:56:23-rados-wip-yuri4-testing-2023-02-07-0246-pacific-distro-default-smithi/7152654

description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools
conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down
pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no
3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}}

2023-02-07T04:37:37.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap...
2023-02-07T04:37:37.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 23b79dd8-a6a0-11ed-bc20-001a4aab830c -- ceph mon dump -f json
2023-02-07T04:37:38.525 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:38 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: cluster 2023-02-07T04:37:37.090670+0000 mgr.smithi052.vjyluh (mgr.14162) 171 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2023-02-07T04:37:39.277 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: cephadm 2023-02-07T04:37:38.365113+0000 mgr.smithi052.vjyluh (mgr.14162) 172 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi052 ...
2023-02-07T04:37:39.277 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager
2023-02-07T04:37:39.277 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0...
2023-02-07T04:37:39.277 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:32:47Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:33:24Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:34:32Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.278 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2023-02-07T04:37:39.279 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus
2023-02-07T04:37:39.279 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0...
2023-02-07T04:37:39.280 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:35:46Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.280 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:36:23Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.280 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr time="2023-02-07T04:37:00Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 
2023-02-07T04:37:39.280 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
2023-02-07T04:37:39.280 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: Traceback (most recent call last):
2023-02-07T04:37:39.281 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in <module>
2023-02-07T04:37:39.281 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     main()
2023-02-07T04:37:39.281 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main
2023-02-07T04:37:39.281 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     r = ctx.func(ctx)
2023-02-07T04:37:39.281 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image
2023-02-07T04:37:39.282 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     return func(ctx)
2023-02-07T04:37:39.282 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy
2023-02-07T04:37:39.282 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     uid, gid = extract_uid_gid_monitoring(ctx, daemon_type)
2023-02-07T04:37:39.282 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring
2023-02-07T04:37:39.282 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus'])
2023-02-07T04:37:39.283 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/var/lib/ceph/23b79dd8-a6a0-11ed-bc20-001a4aab830c/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid
2023-02-07T04:37:39.283 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     raise RuntimeError('uid/gid not found')
2023-02-07T04:37:39.283 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: RuntimeError: uid/gid not found
2023-02-07T04:37:39.283 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: Traceback (most recent call last):
2023-02-07T04:37:39.283 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection
2023-02-07T04:37:39.284 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     yield (conn, connr)
2023-02-07T04:37:39.284 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm
2023-02-07T04:37:39.284 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]:     code, '\n'.join(err)))
2023-02-07T04:37:39.284 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi052 ...
2023-02-07T04:37:39.287 INFO:journalctl@ceph.mon.smithi052.smithi052.stdout:Feb 07 04:37:39 smithi052 ceph-23b79dd8-a6a0-11ed-bc20-001a4aab830c-mon.smithi052[103519]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi052 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager