Project

General

Profile

Feature #41073

cephfs-sync: tool for synchronizing cephfs snapshots to remote target

Added by Venky Shankar 11 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Category:
Administration/Usability
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Reviewed:
Affected Versions:
Component(FS):
Labels (FS):
Pull request ID:

Description

Introduce a rsync like tool (cephfs-sync) for mirroring scheduled and temp cephfs snapshots to sync targets. sync targets could be ceph filesystem on a remote site or even a traditional filesystem. We may want to sync to remote ceph filesystem initially.

We would start with scanning the filesystem based on rctime and feed the file list to rsync. Later, implement (or provide an option) to switch using various sync mechanisms that are much more efficient (such as identifying renames and propagating that to the remote target rather than a copy and delete).


Related issues

Related to fs - Feature #41074: pybind/mgr/volumes: mirror (scheduled) snapshots to remote target New

History

#1 Updated by Venky Shankar 11 months ago

  • Related to Feature #41074: pybind/mgr/volumes: mirror (scheduled) snapshots to remote target added

#2 Updated by Patrick Donnelly 11 months ago

  • Assignee set to Daniel Oliveira

#3 Updated by Patrick Donnelly 5 months ago

  • Target version deleted (v15.0.0)

#4 Updated by Patrick Donnelly 4 months ago

  • Assignee deleted (Daniel Oliveira)
  • Target version set to v16.0.0

#5 Updated by Milind Changire about 2 months ago

  • Assignee set to Milind Changire

#6 Updated by Milind Changire about 2 months ago

Feature Requirements:
  • tool should be implemented as a mgr plugin
  • tool should mirror User Defined as well as Scheduled Snapshots
  • tool should optimize on number of snapshot processes running concurrently
  • tool should run sync process in a containerized environment
  • tool should verify which snapshots are pending and which are completed
  • tool should interact with the snapshot retention mechanism to block purging of snapshots which haven't been synced / mirrored yet
  • sync process should transfer data over a secure connection eg. ssh tunnel
  • sync process should limit use of network bandwidth for data transfers

Also available in: Atom PDF