Ceph » Orchestrator

The following orchestrator commands and features must be available in the Rook Orchestrator:

ceph orch device ls

We have decided that this command must return the list of unused k8s PVs not bound to any PVC. All these PVs uses the same Storage Class.
The name of this Storage Class used for PVs must be stored in the Ceph configuration database:

1. ceph config set mgr mgr/rook/storage_class <storage_class_name>

ceph orch apply -i <drive_group_file>

This will create OSDs following drive group specification, the main difference with cephadm orchestrator is cephadm "devices available" are Rook "PVs available".

Drive group specification have lot of possibilities that are not possible (because intrinsic nature of k8s and because the possibilities to configure OSDs in the Ceph cluster CRD used by the Rook Operator)

So we must support the following drive group attributes:

-encrypted
-data_devices
-db_devices
-wal_devices
-filter_logic

the minimum set of drive group filter attributes supported in mgr/rook to select PVs for must be:

-size
-all
-limit

Probably we will need to add a new attribute to the drive group specification.
storage_class:
We do not have attributes to select specific fast devices .. so this attribute will allow us to select PVs using a certain storage class, this will allow us to use the right PVs for OSDs db/wal components.

ceph orch ls
Must list the OSd service created using drive groups

ceph orch osd rm <osd_id(s)> [--replace] [--force]
Must remove the OSD (OSd list provided) with the sae behavior we have in the cephadm orchestrator

Documentation:
cephadm OSD management: https://docs.ceph.com/en/latest/cephadm/osd/#
OSD Bluestore configuration reference: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#
Rook: Using storage devicesets: https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#using-storageclassdevicesets
Rook: Dedicated metadata and wal device for OSD on PVC: https://github.com/rook/rook/blob/master/Documentation/ceph-cluster-crd.md#dedicated-metadata-and-wal-device-for-osd-on-pvc