Ceph » CephFS

"We may need to introduce a second locking layer to deal with this, that covers draining out all VFS requests before we do our own invals...blech."

By then isn't it too late? By the time we drain and start our invalidate, another read may have already hit the vfs layer (acquiring the lock). I must be missing something. How this handled by other fuse modules?

Maybe; we didn't think this through much beyond going "yep, that's broken".

However, I think we can queue up the invalidate, but not block on it completing, then satisfy any reads that come in from the VFS until the invalidate completes. That satisfies our requirements, and while a malicious user could prevent us from invalidating that might be the way it has to be to satisfy all the invariants involved.

Hmm, you're right, this is a more fundamental problem.

The invalidate is queued in a separate thread, and when we call the invalidate, we don't have the client lock held. So other requests should be getting processed while the invalidate callback is outstanding.

That said, we haven't tested that functionality significantly yet - although I'm surprised we haven't seen a deadlock show up in the nightlies, since they've been running with the invalidate callback enabled for a while now.

It's not any of our internal locking that are getting stuck; it's the VFS inode mutexes in combination with us. If I understand the order correctly:
1) VFS starts reading, locking pages as it goes
2) we have to invalidate and do the upcall
3) upcall to invalidate hits locked pages and waits
4) VFS read needs to call down into ceph-fuse to retrieve more pages (and doesn't drop the page locks, because it's still serving the read)
5) ceph-fuse won't serve up pages because we're trying to drop caps.
6) we are stuck.

We haven't seen this in our nightlies because we aren't generally doing invalidates — the only reason they'll happen on a recently-active file is if there are multiple writers, which none of our regular tests do. (Thus trying to fix up the ior things so we can run them regularly!)

Sage is discussing this on fuse-devel and at LSF/MM (did I get that acronym right?).

The suggestion from Maxim is to modify fuse to serialize reads and invalidate via a mutex. That ought to do the trick.. will need to play with it.

There is also a bunch of fuse writeback (!) stuff that is going to land soonish, so the picture may get a bit more complicated then!

pushed wip-fuse to ceph-client.git