Ceph » Orchestrator

Target: to have same functionality we have in cephadm but in k8s.

In the Ceph orchestrator we assign labels only to hosts:

# ceph orch host label add my_hostname my_label

also we have the possibility to remove labels in the host:

# ceph orch host label rm my_hostname my_label

And we use these labels in placement specification for deploying daemons in certain hosts:

# ceph orch apply <service_type> --placement="label:mylabel" 

or inside yaml specifications:

service_type: <service_type>
placement:
  label: "mylabel" 

Tasks to do

1. Assign/remove labels to k8s hosts using "ceph orch host label" commands:
In k8s we can use:
Add label:

   # kubectl label node k8s-master-00.rook ceph.orchestrator/role=mon

Remove label:

   # kubectl label node k8s-master-00.rook ceph.orchestrator/role-

NOTE: ceph orchestrator label:
"<key>:<value>"
is translated into k8s label:
"ceph.orchestrator/<key>:<value>"

2. The orchestrator command "ceph orch host ls" must show also the labels in the hosts with prefix "ceph.orchestrator"

3. Convert "orchestrator placement specifications" in k8s "pod affinity terms":

Example:

Orchestrator placement specification:

service_type: mon
placement:
  role: "mon" 

or

ceph orch aply mon --placement="role:mon" 

Must be transformed in:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: ceph.orchestrator/role
          operator: In
          values:
          - mon  

Depending of the type of daemon affected by the placement, we need to act in different ways:

For mon, mgr, and osds:
- we need to patch the Ceph cluster CRD with the k8s placement term obtained.
see https://rook.github.io/docs/rook/master/ceph-cluster-crd.html#placement-configuration-settings
- The patch must happen in the right site ( placement-mgr/mon/osd)
see https://rook.github.io/docs/rook/master/ceph-cluster-crd.html#node-affinity

For other daemons, we need to patch the specific CRD (update) or include the affinity term in the yaml file(creation)

RGW: object.yaml
filesystem: filesystem.yaml
NFS: nfs.yaml

Note: Apply the orchestrator placement will override rook operator default placements