Ceph » CephFS

This really seems like the wrong approach to me. Is this callback going to be synchronous or async? Imagine you get a bunch of caps recalled all at once (or the client decides to voluntarily return them for whatever reason). Either you have to issue these invalidations synchronously in some fashion, or you have a window where you've returned caps but ganesha still has cached attributes that it's operating on.

You could mitigate the synchronous nature by creating a way to invalidate them in batches, but that's really complex. If you go with an async notification then that's potentially racy, especially once we start considering tossing pnfs into the mix.

I think it would be better to not have ganesha cache anything by default between RPCs. The exception there would be to allow it to cache things when it has a delegation.

Now, that said...Frank was doing some experimentation with disabling caches on ganesha+ceph last week and saw a perf hit on a readdir test. It's still early days there though, and I think that may just be an indicator that the ceph userland client could use some optimization. The single giant client mutex is pretty gross...

I'm open-minded here if someone can convince me that multiple layers of caching is beneficial, but I'm just not seeing the benefit to the complexity right now. Multi-layer caching problems are a real pain to sort out.

My intent in this experiment was, yes, that upcalls be synchronous. To be brief I'd say that my understanding was, if nfs-ganesha has state that must be invalidated relative to, e.g., a CAP being revoked, the invalidation must be synchronous--but you've already paid for the synchrony in the protocol. I didn't further elaborate this (e.g., bulk calls) because I am also one of the advocates for permitting fsals like ceph and rgw to bypass the ganesha cache. Basically, that's what MDCACHE is for. I'm surprised that we get benefit from even the dirent cache as it's presently implemented.

iirc, we still have the file handle cache, even if we disable caching in general, right?

Is it the case that we would still need the invalidation mechanism for file handles? My knowledge of NFS internals is insufficient to know how large the "files with file handles" population is likely to be, and so whether we will care about doing cache invalidation to enable the client to evict inodes (at mds request) which have cached file handles in the ganesha layer.

Jeff Layton wrote:

I looked at this, but I think the real solution to this problem is to just prevent ganesha from caching these objects as much as we can. libcephfs can still cache aggressively when we do that. We just end up reconstituting the ganesha objects from the libcephfs caches when they have the right caps.

We're already recommending that configuration on ganesha going forward, so I think we can just say WONTFIX here.

Most of the scenarios can be solved with the default configuration, but if there are recursive queries, such as ls -lR large directory, rm -rf, etc., it will be easy to raise the nfs-ganesha cache to a very high level, and cause mds trim to fail. And eventually cause the node cpu to run out