Project

General

Profile

Feature #18537

libcephfs cache invalidation upcalls

Added by John Spray about 7 years ago. Updated over 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Component(FS):
Client, Ganesha FSAL
Labels (FS):
Pull request ID:

Description

Matt Benjamin did some work in this area:

https://github.com/linuxbox2/nfs-ganesha/tree/ceph-invalidates
https://github.com/linuxbox2/ceph/tree/invalidate

The goal is to enable Client instances inside ganesha daemons to respond properly to requests to relinquish caps send from the MDS.


Related issues

Related to CephFS - Feature #12334: nfs-ganesha: handle client cache pressure in NFS Ganesha FSAL Resolved

History

#1 Updated by Jeff Layton about 7 years ago

This really seems like the wrong approach to me. Is this callback going to be synchronous or async? Imagine you get a bunch of caps recalled all at once (or the client decides to voluntarily return them for whatever reason). Either you have to issue these invalidations synchronously in some fashion, or you have a window where you've returned caps but ganesha still has cached attributes that it's operating on.

You could mitigate the synchronous nature by creating a way to invalidate them in batches, but that's really complex. If you go with an async notification then that's potentially racy, especially once we start considering tossing pnfs into the mix.

I think it would be better to not have ganesha cache anything by default between RPCs. The exception there would be to allow it to cache things when it has a delegation.

Now, that said...Frank was doing some experimentation with disabling caches on ganesha+ceph last week and saw a perf hit on a readdir test. It's still early days there though, and I think that may just be an indicator that the ceph userland client could use some optimization. The single giant client mutex is pretty gross...

I'm open-minded here if someone can convince me that multiple layers of caching is beneficial, but I'm just not seeing the benefit to the complexity right now. Multi-layer caching problems are a real pain to sort out.

#2 Updated by Matt Benjamin about 7 years ago

My intent in this experiment was, yes, that upcalls be synchronous. To be brief I'd say that my understanding was, if nfs-ganesha has state that must be invalidated relative to, e.g., a CAP being revoked, the invalidation must be synchronous--but you've already paid for the synchrony in the protocol. I didn't further elaborate this (e.g., bulk calls) because I am also one of the advocates for permitting fsals like ceph and rgw to bypass the ganesha cache. Basically, that's what MDCACHE is for. I'm surprised that we get benefit from even the dirent cache as it's presently implemented.

#3 Updated by John Spray about 7 years ago

iirc, we still have the file handle cache, even if we disable caching in general, right?

Is it the case that we would still need the invalidation mechanism for file handles? My knowledge of NFS internals is insufficient to know how large the "files with file handles" population is likely to be, and so whether we will care about doing cache invalidation to enable the client to evict inodes (at mds request) which have cached file handles in the ganesha layer.

#4 Updated by John Spray about 7 years ago

  • Assignee set to Jeff Layton

#5 Updated by Patrick Donnelly almost 6 years ago

  • Related to Feature #12334: nfs-ganesha: handle client cache pressure in NFS Ganesha FSAL added

#6 Updated by Jeff Layton about 5 years ago

  • Status changed from New to Rejected

I looked at this, but I think the real solution to this problem is to just prevent ganesha from caching these objects as much as we can. libcephfs can still cache aggressively when we do that. We just end up reconstituting the ganesha objects from the libcephfs caches when they have the right caps.

We're already recommending that configuration on ganesha going forward, so I think we can just say WONTFIX here.

#7 Updated by Patrick Donnelly about 5 years ago

  • Category deleted (109)
  • Component(FS) Client, Ganesha FSAL added
  • Component(FS) deleted (libcephfs)

#8 Updated by huanwen ren over 4 years ago

Jeff Layton wrote:

I looked at this, but I think the real solution to this problem is to just prevent ganesha from caching these objects as much as we can. libcephfs can still cache aggressively when we do that. We just end up reconstituting the ganesha objects from the libcephfs caches when they have the right caps.

We're already recommending that configuration on ganesha going forward, so I think we can just say WONTFIX here.

Most of the scenarios can be solved with the default configuration, but if there are recursive queries, such as ls -lR large directory, rm -rf, etc., it will be easy to raise the nfs-ganesha cache to a very high level, and cause mds trim to fail. And eventually cause the node cpu to run out

Also available in: Atom PDF