Tasks #51341
Updated by Ramana Raja almost 3 years ago
In certain rare cases, all the Ceph Monitors might end up with corrupted Monitor stores. The Monitor stores can be recovered from the OSDs using the monmap tool, and Monitors can be brought back online. MDSMaps however are lost. Additional steps are required to bring back the file system(s) and the MDSs. Steps for recovery will differ between single active MDS and multi active MDS file systems, single and multi file systems in a Ceph cluster The steps identified to bring back a single active MDS file system post recovery of Monitor stores and Monitor is as follows: - Ensure all MDSs are stopped on cluster systemctl stop ceph-mds@<mds-id> - Force create ceph file system using existing file system pools ceph fs new <fs-name> <cephfs-metadata-poolname> <cephfs-data-poolname> --force - Reset file system ceph fs reset <fs-name> --yes-i-really-mean-it - Restart MDSs systemctl start ceph-mds@<mds-id> For multi MDS file system, it may be possible to recover by marking the new file system unjoinable, setting max_mds, and then marking it joinable.