Ceph

The situation is as follows: I have a Ceph cluster that initially served as RBD block storage for OpenStack. Later, I created CephFS with metadata and data pools using a triple replication strategy. The OSDs (70 HDDs and 20 SSDs) were reused from the original configuration. The metadata pool is associated with SSD-type OSDs, while the data pool is associated with HDD-type OSDs.

However, as the CephFS file count explosively increased, several issues emerged. First, there was a concern about the low storage space utilization (1/3), resulting in excessively high storage costs. Second, the CephFS object count surged, causing both the metadata pool and data pool object counts to exceed 20 million. The OSD service startup time increased from less than a minute to the current 20 minutes, making maintenance challenging.

Therefore, I am considering separating CephFS from RBD. I plan to store the metadata pool in a new, dedicated SSD pool and create a new HDD-based data pool using the Erasure Code type. Currently, there are some challenges, and I would like your assistance in exploring better solutions for this problem:

The Ceph version is v15.2.13. When attempting to create a second File System, a prompt indicates that multi-cephfs is an experimental feature, and data loss may occur, which we are hesitant to use.
We plan to use the "add_data_pool" method to add an Erasure Code data pool. The original data will be copied to the new EC pool using the rsync tool, with writes stopped before the copy.
According to the crush rule, we intend to isolate the new SSDs from the original ones, either by adding a new root or device_class strategy. Then, we will associate the original metadata pool with the new crush rule.
Especially in the third point, there is a possibility that the original CephFS may become completely unusable, and predicting downtime is challenging.