Ceph » CephFS

David and I have posted comments on github about the fix to allow multiple
clients opening the same file to get a valid fd (instead of EEXIST). The
summary is that the changes made to wip-3625 successfully resolve the EEXIST
problem, but introduce another race with permissions checking, because we are
only checking permissions on open if the file already exists. With two
clients that both send create requests, one will perform the create at the
mds, and the other will perform an open, but both clients will think that
they've created the file, and skip the permissions checking. There are a few
ways to resolve this:

1. Send the create request with O_EXCL, and if it returns EEXIST, send a
lookup request.

2. Do the permissions checking at the mds. The problem with this currently
is that we don't send the secondary group ids to the mds on the create
request, so its permissions checks can currently only check against uid and
gid. We could add the secondary group ids to the create request to resolve
this.

3. Send back a created flag with the reply. We don't have request specific
reply structures, so this seems non-trivial.

4. Ignore this for now, assuming that permissions checking is handled
properly for existing files, so if we end up in the create path, we are in
the multi-client create race, and can assume they will all open the file with
the same permissions. This isn't really posix compliant, and clients could
perform an open/create with:

open(foo, O_CREAT|O_RDONLY, 0400);

in which case, posix requires that the client that wins the create gets a
valid fd, while the other clients get EACCES. If we ignore it, it means that
someone could gain access to a file they don't have permission to access by
timing an open of the file at the same time that its being created.

From Sage:

Hrm. I like #3 the best. Although we probably also need to do #2.
Probably. Here's the thing:

We will never enforce all security at the MDS; the whole premise is that
we can delegate cache consistency to the client and do things there. But:
we also want to get the semantics right (here it might help). And more
importantly, we want ot support a mode where the client is only allowed to
be a specific user (e.g., a libcephfs user with a cephx capabitilty of
uid=1001). I suspect that is a big project, though, which a whole host of
issues that we don't want to get into right now.

Which leads me back to 2 being the cleanest approach for this particular
problem (where we just want to get the semantics right).

1 might work if we can coax the mds into including the dentry+inode in
the EEXIST reply... in that case the subsequent lookup/open will be 100%
client-side and performance won't be any worse.

I know you guys did a couple rounds on this one, what's the status?

The kernel side has been reviewed and tested, but needs to be merged. The fuse side has been tested, but I think it still needs to be reviewed (and merged).