cephfs-sync: tool for synchronizing cephfs snapshots to remote target
Introduce a rsync like tool (cephfs-sync) for mirroring scheduled and temp cephfs snapshots to sync targets. sync targets could be ceph filesystem on a remote site or even a traditional filesystem. We may want to sync to remote ceph filesystem initially.
We would start with scanning the filesystem based on rctime and feed the file list to rsync. Later, implement (or provide an option) to switch using various sync mechanisms that are much more efficient (such as identifying renames and propagating that to the remote target rather than a copy and delete).
#6 Updated by Milind Changire 8 months ago
- tool should be implemented as a mgr plugin
- tool should mirror User Defined as well as Scheduled Snapshots
- tool should optimize on number of snapshot processes running concurrently
- tool should run sync process in a containerized environment
- tool should verify which snapshots are pending and which are completed
- tool should interact with the snapshot retention mechanism to block purging of snapshots which haven't been synced / mirrored yet
- sync process should transfer data over a secure connection eg. ssh tunnel
- sync process should limit use of network bandwidth for data transfers