Actions
Bug #63101
opencephadm seems to be trying to pull images named after daemons
Status:
New
Priority:
Normal
Assignee:
-
Category:
orchestrator
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Attempting to upgrade from quincy 17.2.5 using
ceph orch upgrade start --ceph-version 18.2.0(as described here: https://docs.ceph.com/en/quincy/cephadm/upgrade/) results in cephadm trying to pull an image based on the name of a daemon, rather than the image the daemon is derived from:
Here's an example of the trouble, from /var/log/ceph/cephadm.log:
Note this line:
INFO Pulling container image osd.60...
2023-10-04 09:02:41,109 7fc94cf59b80 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'osd.60', 'pull'] 2023-10-04 09:02:41,319 7fc94cf59b80 INFO Pulling container image osd.60... 2023-10-04 09:02:43,403 7fc94cf59b80 INFO Non-zero exit code 125 from /usr/bin/podman pull osd.60 2023-10-04 09:02:43,404 7fc94cf59b80 INFO /usr/bin/podman: stderr Resolving "osd.60" using unqualified-search registries (/etc/containers/registries.conf) 2023-10-04 09:02:43,404 7fc94cf59b80 INFO /usr/bin/podman: stderr Trying to pull quay.io/osd.60:latest... 2023-10-04 09:02:43,404 7fc94cf59b80 INFO /usr/bin/podman: stderr Trying to pull registry.access.redhat.com/osd.60:latest... 2023-10-04 09:02:43,404 7fc94cf59b80 INFO /usr/bin/podman: stderr Trying to pull registry.redhat.io/osd.60:latest... 2023-10-04 09:02:43,404 7fc94cf59b80 INFO /usr/bin/podman: stderr Trying to pull docker.io/library/osd.60:latest... 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr Error: 4 errors occurred while pulling: 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr * initializing source docker://quay.io/osd.60:latest: reading manifest latest in quay.io/osd.60: StatusCode: 404, <!doctype html> 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr <html lang=en> 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr <title>404 Not Foun... 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr * initializing source docker://registry.access.redhat.com/osd.60:latest: reading manifest latest in registry.access.redhat.com/osd.60: name unknown: Repo not found 2023-10-04 09:02:43,405 7fc94cf59b80 INFO /usr/bin/podman: stderr * initializing source docker://registry.redhat.io/osd.60:latest: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication 2023-10-04 09:02:43,406 7fc94cf59b80 INFO /usr/bin/podman: stderr * initializing source docker://osd.60:latest: reading manifest latest in docker.io/library/osd.60: errors: 2023-10-04 09:02:43,406 7fc94cf59b80 INFO /usr/bin/podman: stderr denied: requested access to the resource is denied 2023-10-04 09:02:43,406 7fc94cf59b80 INFO /usr/bin/podman: stderr unauthorized: authentication required 2023-10-04 09:02:43,406 7fc94cf59b80 INFO /usr/bin/podman: stderr 2023-10-04 09:02:43,406 7fc94cf59b80 DEBUG Pulling image for `command_pull` failed: Failed to pull container image. Check that host(s) are logged into the registry 2023-10-04 09:02:43,406 7fc94cf59b80 ERROR ERROR: Failed to pull container image. Check that host(s) are logged into the registry
This will go on forever, even after issuing
ceph orch upgrade stop
I tried downloading the specific image with
podman pull quay.io/ceph/ceph:v18.2.0but it didn't have an effect on the behavior.
At this point, osd.60 is marked
autoout,exists, which I presume was done by ceph orchestrator. What I suspect is supposed to happen is cephadm is told to pull ceph:v18.2.0 and rebuild each daemon. Is that right?
Actions