Ceph » CephFS

Currently, this is how we deal with hardlinks vs. layouts specifying non-default data pools: files in the non-default pools write an extra backtrace to the default data pool, so that anyone resolving them via a hardlink can do so easily (or anyone resolving an NFS file handle).

This is pretty bad in IOPS terms in the case where all your files are in non-default data pools, or in cases where you perhaps want to remove the default data pool (pools are expensive because pools=PGs.)

• We could have a per-data-pool flag where certain pools are un-hard-linkable (and potentially un-NFS-able). If a file is created in that pool, don't add the default pool to old_pools at creation time. If someone tries to hard link to that inode, give them an error. We probably can't detect NFS mounts though, so that would rely on user sanity...
• We could extend the client protocol to allow clients (like NFS exporters) to specify a hint about what data pool to look up inodes in. Possibly this could just be from the layout of whatever they're mounting as root: if someone is doing an NFS export that only touches a single non-default-pool-layout subtree, then we know to do hardlink lookups within that pool, and we can avoid the need for the default pool backtraces. This also requires forbidding cross-client-root hardlinks, which is obviously not possible to do in advance: this would only work for certain carefully controlled client cases. However, when we're controlling NFS ourselves (e.g. Manila) we could do it carefully enough to enable this method.
• We could do this on a per-inode basis: when someone tries to hardlink, see if old_pools already contains the default pool. If it doesn't, then insist on modifying the linked-to inode's old_pools to contain the default data pool before allowing the hardlink creation. Simiarly, for NFS, clients could have a flag that causes the default-pool-backtrace to be created before a file is opened (and probably on creation too if created by an NFS client).