Ceph » CephFS

An oddity was reported in the thread "MDS has inconsistent performance" on ceph-devel, starting at http://www.spinics.net/lists/ceph-devel/msg21987.html. The user noticed that when clients were doing a file create workload in their own directory, they would start out sending only creates to the MDS, but that eventually they switched to doing a prior lookup for every create. This of course was because the client was losing its I_COMPLETE flag on the directory.

Investigation revealed that I_COMPLETE was getting cleared because the client did a create while holding the Fs cap (but not Fx) on the directory, so it dropped the Fs cap as part of the create request, and then when it got re-assigned in the response it had to clear the flag. This was odd since the client had previously held Fsx on the directory.

The drop of Fx turned out to be the result of Client::check_caps(). check_caps() was invoked when the MDS revoked some unrelated caps, and check_caps() calls send_cap(), which drops all caps which aren't included in a "retain" parameter. check_caps is filling out retain with the combined output of Inode::caps_wanted() and the caps used on the inode.
Unfortunately, caps_wanted() only returns anything at all if the inode is opened, based on the mode in use. Directories are basically never opened, and their caps aren't used much (ever?) either.