Bug #52812


stat->fsid is not unique among filesystems exported by the ceph server

Added by Sachin Prabhu over 2 years ago. Updated almost 2 years ago.

Target version:
% Done:


3 - minor
Affected Versions:
Crash signature (v1):
Crash signature (v2):


We are working on a new kubernetes operator to export ceph filesystem using samba. To do this, we mount the filesystem across multiple nodes and use ctdb to allow us to setup an active-active cluster. One of the issues we were investigating was the issue of filehandles(file_id) generated by Samba. It is critical that the filehandle is consistent on all the nodes of the cluster. The file handles are usually generated using the device id and the inode number of the file.

The deviceid is obtained from the stat() call with return value stat->st_dev. On ceph-kmod, this value is eventually obtained from get_anon_bdev() which generates a virtual block device with a generated device id. This can be different on different nodes and will depend on the order in which the filesystems were mounted. We therefore cannot use the device id to generate the filehandle since we cannot guaranty that the device id across the nodes will be the same.

We were therefore investigating the fsid obtained with a statfs() which returns statfs->f_fsid. We can use this with the samba vfs_plugin vfs_fileid to generate consistent filehandles across the cluster. This is set at
static int ceph_statfs(struct dentry *dentry, struct kstatfs *buf)

        fsid = le64_to_cpu(*(__le64 *)(&monc->monmap->fsid)) ^
               le64_to_cpu(*((__le64 *)&monc->monmap->fsid + 1));
        buf->f_fsid = u64_to_fsid(fsid);

But since the monc->monmap->fsid is the cluster id, we may need an additional component to differentiate between the multiple file systems exported from the cluster.
Without this change, we may have two different files on two different filesystems exported from the same cluster end up with the same filehandle which could lead to problems on the Samba file server.

Related issues 1 (0 open1 closed)

Copied to CephFS - Bug #53045: stat->fsid is not unique among filesystems exported by the ceph serverResolvedJeff Layton

Actions #1

Updated by Sachin Prabhu over 2 years ago

The fileid is an internal representation of the file in the samba file server. This is used to identify a file internally.

It is represented by
struct file_id {
uint64_t devid;
uint64_t inode;
uint64_t extid;
}/* [public] */;
of which extid is unused by default.

When a file is opened by a client and is provided with an oplock/lease/byte range lock, the samba server records this in an internal database using the fileid as the key. When another client attempts to open the file, the server first checks the internal database for any existing record for the file.

CTDB provides the clustered database required to operate the samba filesystem in an active-active setup on a cluster of nodes. The operation is similar as described above except that the database now use is a cluster database and the records could be available on any of the nodes in the cluster.

It is therefore vital that the file_id uniquely describes a file in the cluster and is also consistent across the cluster.

By default it uses the device id which cannot be used here as described in the summary. We are therefore looking at using fsid. For a cluster exporting a single filesystem, the issue described by the summary is not a problem. The problem will be seen when multiple filesystems with the same fsid are exported by the clustered fileserver.

Actions #3

Updated by Jeff Layton over 2 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Jeff Layton
Actions #4

Updated by Jeff Layton over 2 years ago

  • Copied to Bug #53045: stat->fsid is not unique among filesystems exported by the ceph server added
Actions #5

Updated by Jeff Layton almost 2 years ago

  • Status changed from Fix Under Review to Resolved
Actions #6

Updated by Jeff Layton almost 2 years ago

  • Description updated (diff)

Also available in: Atom PDF