Ceph » CephFS

Right now re-adding a file system as a peer to another file system requires that the cluster admin/operator cleanup synchronized snapshots: See https://docs.ceph.com/en/latest/cephfs/cephfs-mirroring/#re-adding-peers

Also, it is recommended to purge synchronized directories from the peer before re-adding it to another file system (especially those directories which might exist in the new primary file system). This is not required if re-adding a peer to the same primary file system it was earlier synchronized from.

This requires that the cluster admin purges synchronized snapshots (for directory paths that were added for snap mirroring) which is a cumbersome thing to do.

To mitigate this, the mirror daemon should be able to identify a re-peered file system and cleanup existing snapshots for directory paths that have been added for mirroring before starting further snap syncs. One way to do this is to start marking the directory path of the remote file system with an xattr (maybe reuse ceph.mirror.info which currently is constrained to root and is used to prevent multiple file systems adding it as a peer). The root directory is marked with ceph.mirror.info xattr when its added a peer and is removed when the file system is removed as a peer. When the file system is peered to a different file system (same or different cluster), the ceph.mirror.info xattr on root and on the directory paths would mismatch, signifying that the snapshots under the directory were synchronized when the file system was a peer to a different cluster/file system.