Ceph » Orchestrator

The cephadm(v16.2.7) bootstrap by default chooses container images quay.io/ceph/ceph:v16 and docker.io/ceph/daemon-base:latest-pacific-devel. Since we want to avoid using devel and latest container images in production, we pulled the required images (different tags) prior to running bootstrap. Also, we mentioned the image name and --skip-pull parameter in bootstrap command.
Still cephadm uses the image docker.io/ceph/daemon-base:latest-pacific-devel for many of the daemons, and it is still pulling the image even though --skip-pull is mentioned.

Due to this, daemons on different host's are running on different versions of container images.

Hence, there is no provision to use specific image instead of docker.io/ceph/daemon-base:latest-pacific-devel during bootstrap for consistency across all daemons in the cluster.

How to reproduce it :

podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/ceph/daemon-base v6.0.6-stable-6.0-pacific-centos-8-x86_64 4e3c927255bb 7 weeks ago 1.24 GB
quay.io/ceph/ceph v16.2.7 cc266d6139f4 7 weeks ago 1.24 GB

sudo cephadm --image quay.io/ceph/ceph:v16.2.7 bootstrap --skip-monitoring-stack --mon-ip ... --cluster-network ... --ssh-user ceph_user --config /home/ceph_user/ceph_bootstrap/ceph.conf --initial-dashboard-password Q5446UBS3KK9 --dashboard-password-noupdate --no-minimize-config --skip-pull

Below are some entries from cephadm.log, which clearly shows its trying to pull image even --skip-pull is provided.

2022-01-27 17:11:13,900 7f01b6621b80 INFO Deploying mon service with default placement...
2022-01-27 17:11:14,212 7f211cc85b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific-devel', 'ls']
2022-01-27 17:11:14,296 7f211cc85b80 DEBUG /bin/podman: 3.3.1
2022-01-27 17:11:14,660 7f211cc85b80 DEBUG /bin/podman: 4da6ea847240,24.26MB / 134.9GB
2022-01-27 17:11:14,660 7f211cc85b80 DEBUG /bin/podman: 52b12ff050d8,390.7MB / 134.9GB
2022-01-27 17:11:14,660 7f211cc85b80 DEBUG /bin/podman: 5c979c84d182,4.342MB / 134.9GB
2022-01-27 17:11:14,766 7f211cc85b80 DEBUG systemctl: enabled
2022-01-27 17:11:14,778 7f211cc85b80 DEBUG systemctl: active
2022-01-27 17:11:14,912 7f211cc85b80 DEBUG /bin/podman: 52b12ff050d88841131aa6508f7576a1dca8e0004db08384dd13dca6c2d3b725,quay.io/ceph/ceph:v16.2.7,cc266d6139f4d044d28ace2308f7befcdfead3c3e88bc3faed905298cae299ef,2022-01-27 17:10:33.135056074 +0530 IST,
2022-01-27 17:11:15,059 7f211cc85b80 DEBUG /bin/podman: [quay.io/ceph/ceph@sha256:2f7f0af8663e73a422f797de605e769ae44eb0297f2a79324739404cc1765728 quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54]
2022-01-27 17:11:15,456 7f01b6621b80 DEBUG /usr/bin/ceph: Scheduled mon update...
2022-01-27 17:11:15,641 7f211cc85b80 DEBUG /bin/podman: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)

2022-01-27 17:11:15,972 7f01b6621b80 INFO Deploying mgr service with default placement...
2022-01-27 17:11:16,127 7f211cc85b80 DEBUG systemctl: enabled
2022-01-27 17:11:16,140 7f211cc85b80 DEBUG systemctl: active
2022-01-27 17:11:16,296 7f211cc85b80 DEBUG /bin/podman: 4da6ea847240bab786f596ddc87160e11056c74aa7004dc38ee12be331a5ea4e,quay.io/ceph/ceph:v16.2.7,cc266d6139f4d044d28ace2308f7befcdfead3c3e88bc3faed905298cae299ef,2022-01-27 17:10:25.830630277 +0530 IST,
2022-01-27 17:11:17,023 7f0b0c505b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific-devel', 'ceph-volume', '--fsid', 'e3c9bff6-7f65-11ec-bdff-0015171590ba', '--', 'inventory', '--format=json-pretty', '--filter-for-batch']
2022-01-27 17:11:17,102 7f0b0c505b80 DEBUG /bin/podman: 3.3.1
2022-01-27 17:11:17,275 7f0b0c505b80 DEBUG /bin/podman: 4da6ea847240,24.71MB / 134.9GB
2022-01-27 17:11:17,275 7f0b0c505b80 DEBUG /bin/podman: 52b12ff050d8,390.8MB / 134.9GB
2022-01-27 17:11:17,275 7f0b0c505b80 DEBUG /bin/podman: d242f1fa7a66,28.28MB / 134.9GB
2022-01-27 17:11:17,417 7f0b0c505b80 INFO Inferring config /var/lib/ceph/e3c9bff6-7f65-11ec-bdff-0015171590ba/mon.hcictrl01/config
2022-01-27 17:11:17,417 7f0b0c505b80 DEBUG Using specified fsid: e3c9bff6-7f65-11ec-bdff-0015171590ba
2022-01-27 17:11:17,620 7f01b6621b80 DEBUG /usr/bin/ceph: Scheduled mgr update...
2022-01-27 17:11:17,727 7f0b0c505b80 DEBUG stat: Trying to pull docker.io/ceph/daemon-base:latest-pacific-devel...

2022-01-27 17:11:18,489 7f01b6621b80 INFO Deploying crash service with default placement...
2022-01-27 17:11:18,763 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:18,768 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:18,774 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:18,779 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:18,784 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:18,789 7f3ed21eeb80 DEBUG sestatus: SELinux status: disabled
2022-01-27 17:11:19,434 7f75157f1b80 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific-devel', 'deploy', '--fsid', 'e3c9bff6-7f65-11ec-bdff-0015171590ba', '--name', 'mgr.hcictrl01.njkjzk', '--meta-json', '{"service_name": "mgr", "ports": [9283], "ip": null, "deployed_by": ["quay.io/ceph/ceph@sha256:2f7f0af8663e73a422f797de605e769ae44eb0297f2a79324739404cc1765728", "quay.io/ceph/ceph@sha256:bb6a71f7f481985f6d3b358e3b9ef64c6755b3db5aa53198e0aac38be5c8ae54"], "rank": null, "rank_generation": null}', '--config-json', '-', '--tcp-ports', '9283', '--reconfig']
2022-01-27 17:11:19,542 7f75157f1b80 DEBUG /bin/podman: 3.3.1

OS : CentOS Stream release 8
Kernel : Linux hcictrl01 4.18.0-348.2.1.el8_5.x86_64 Improve README #1 SMP
Podman version 3.3.1
Ceph version : Ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)