Ceph » CephFS

After some discussion and agreeing on the approach, below is the proposed design:

Direct Addressing Scheme for Snapshots

Snapshots are tied to subvolumes. This enforces a scheme that requires subvolume access for addressing snapshots. Such a scheme requires searching all subvolume incarnations to lookup a snapshot. To mitigate the cost involved in searching, snapshots are taken at the subvolume directory level (rather than just user data) and include active subvolume metadata. This decouples snapshots from subvolumes. Algorithmically, the cost of searching a snapshot is still O(N), however, most implementations will have the search optimized by performing doing a single filesystem call rather than performing multiple calls one for each subvolume incarnation. Metadata in a snapshot plays a critical role which is detailed in subsequent sections.

Snapshots listing does not involve aggregating snapshots from all incarnations. Again, algorithmically, cost is still O(N), however, most implementations are optimized by performing a single filesystem call. However, subvolume listing still needs to exclude inactive subvolumes.

Maintaining Subvolume Incarnations

UUID directory for the subvolume will be pruned but keeping its metadata intact for future referencing (for snapshots). Implementation wise, metadata file would be prefixed with an incarnation-id. Active subvolumes metadata will have a "pointer" (in the form of a symbolic link) to its metadata file.

/volumes/<group>/sub0/.meta.0
/volumes/<group>/sub0/.meta.1
/volumes/<group>/sub0/<uuid3>
/volumes/<group>/sub0/.meta.2
/volumes/<group>/sub0/.meta  -> ./.meta.2

Metadata Updates

Subvolume snapshot operations need to update subvolume metadata. A subvolume could have been created at an earlier incarnation that current. So, how does a snapshot operations map a snapshot to its subvolume incarnation? Remember, that snapshots include active subvolume metadata -- subvolume metadata in a snapshot points to the incarnation during which the snapshot was created, hence, the incarnation of the subvolume itself. I.e., Subvolume metadata in a snapshot is a wormhole to its subvolume incarnation. This simplifies snapshot operations, especially cloning which had to store the incarnation-id in clone subvolume metadata (C.f., Clone Metadata).

Metadata Garbage Collection

The issue remains of garbage collecting subvolume metadata. Since snapshots are directly addressable, there is no way of knowing when all snapshots for a subvolume incarnation have been deleted. To mitigate this, the subvolume metadata needs to keep a track of active snapshot count. A snapshot delete operation adjusts the active snapshot count for its respective subvolume and prunes metadata when no snapshots exist. Note that metadata purge should only be performed when the snapshot to be deleted not a part of currently active subvolume.

Incarnation-Id Choices

Incarnation-id assigned to subvolume should be distinct each time a subvolume is created. Choices here range from distinct strings, integer values, UUIDs. Implementations can choose whatever fits well to the system. UUIDs are perhaps the best choice for almost all implementations (plus an added benefit of reusing uuid component from subvolume path).

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".