Ceph » CephFS

We are working on a new kubernetes operator to export ceph filesystem using samba. To do this, we mount the filesystem across multiple nodes and use ctdb to allow us to setup an active-active cluster. One of the issues we were investigating was the issue of filehandles(file_id) generated by Samba. It is critical that the filehandle is consistent on all the nodes of the cluster. The file handles are usually generated using the device id and the inode number of the file.

The deviceid is obtained from the stat() call with return value stat->st_dev. On ceph-kmod, this value is eventually obtained from get_anon_bdev() which generates a virtual block device with a generated device id. This can be different on different nodes and will depend on the order in which the filesystems were mounted. We therefore cannot use the device id to generate the filehandle since we cannot guaranty that the device id across the nodes will be the same.

We were therefore investigating the fsid obtained with a statfs() which returns statfs->f_fsid. We can use this with the samba vfs_plugin vfs_fileid to generate consistent filehandles across the cluster. This is set at

static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf)
{
..
        fsid = le64_to_cpu(*(__le64 *)(&monc->monmap->fsid)) ^
               le64_to_cpu(*((__le64 *)&monc->monmap->fsid + 1));
..    
        buf->f_fsid = u64_to_fsid(fsid);
..
}

But since the monc->monmap->fsid is the cluster id, we may need an additional component to differentiate between the multiple file systems exported from the cluster.
Without this change, we may have two different files on two different filesystems exported from the same cluster end up with the same filehandle which could lead to problems on the Samba file server.