Ceph » Orchestrator

I've had this idea kicking around in the back of my mind for a while and a recent mailing list thread got me thinking about it more seriously. The documentation for isolated/disconnected installs is here: https://docs.ceph.com/en/latest/cephadm/install/#deployment-in-an-isolated-environment

In particular steps 3-4 say:

3. Push your container image to your local registry. Here are some acceptable kinds of container images:
   * Ceph container image. See Ceph Container Images.
   * Prometheus container image
   * Node exporter container image
   * Grafana container image
   * Alertmanager container image

4. Create a temporary configuration file to store the names of the monitoring images. (See Using custom images):

cat <<EOF > initial-ceph.conf

[mgr]
mgr/cephadm/container_image_prometheus = *<hostname>*:5000/prometheus
mgr/cephadm/container_image_node_exporter = *<hostname>*:5000/node_exporter
mgr/cephadm/container_image_grafana = *<hostname>*:5000/grafana
mgr/cephadm/container_image_alertmanager = *<hostname>*:5000/alertmanger

These are a bit vague and do not precisely cover what images images are used by ceph or what the exact config keys are for all those images. As a developer I'd be reaching for the source code at this point. That's not a great workflow and if we want to do it in documentation we constnatly have to remember to update docs when we update code.

Rather, I propose the following command(s):
cephadm list-images [--config-template|--brief]

In the default form the command would output a summary (perhaps in YAML) of all images known to cephadm along with some metadata:

  $ cephadm list-images
  ---
  - image: quay.io/ceph/ceph:v18
    role: ceph
    config_key: mgr/cephadm/container_image_base
    default_config_value: quay.io/ceph/ceph
  - image: quay.io/prometheus/prometheus:v2.43.0
    role: monitoring
    config_key: mgr/cephadm/container_image_prometheus
  # ... and so on ... 
  

The option [--brief] would list images only, line by line in a form suitable for shell scripting.

The other form, with the use of `--config-template` would output a ceph.conf style text with the config keys in comments that the user can uncomment and modify

  $ cephadm list-images --config-template
  [mgr]
  # Uncomment and modify the following lines to change the image used for each container
  #mgr/cephadm/container_image_base = quay.io/ceph/ceph
  #mgr/cephadm/container_image_prometheus = quay.io/prometheus/prometheus:v2.43.0
  #mgr/cephadm/container_image_nvmeof = quay.io/ceph/nvmeof:latest
  

After these commands are added the documentation could now read:

3. Push container images to your local registry. The list of images that a cephadm managed cluster may use can be listed using
`cephadm list-images`. Mirror the images you plan to use that listed by that command, for example:

  for image in $(cephadm list-images --brief); do
    skopeo copy "${image}" myreg.example.com:5000/$(echo "${image}" | cut -d/ -f2-)
  done
  

4. Create a configuration template and modify it so that the images listed within point to your registry.

   $ `cephadm list-images --config-template > initial-ceph.conf
   $ $EDITOR initial-ceph.conf
   

Uncomment the lines in the file and change the values to match the images stored your registry:

   [mgr]
   # Uncomment and modify the following lines to change the image used for each container
   mgr/cephadm/container_image_base = myreg.example.com:5000/ceph/ceph
   mgr/cephadm/container_image_prometheus = myreg.example.com:5000/prometheus/prometheus:v2.43.0
   mgr/cephadm/container_image_nvmeof = myreg.example.com:5000/ceph/nvmeof:mycustombuild