Ceph » CephFS

Hmm, I like the basic thrust of this, but I'm a little concerned as well — we have other tickets to let clients reconnect outside of the window. (I hope the reasons for desiring that are obvious — forcing disconnects without allowing some kind of reconnect is a heck of a user problem.) If we're blacklisting the clients this becomes a lot more difficult, because they won't be able to continue OSD ops without reconnecting and losing their identity.

There can be certain cases where a client can reconnect after being evicted, e.g. if:

• the client didn't hold any write capabilities: it can safely be permitted to rejoin the filesystem, although in the interim it might have shown bogus data to userspace if it thought it had a valid read cache of something that we had actually granted elsewhere.
• the caps held by this client were not in demand by anyone else, so we were able to hold onto them and subsequently re-issue them

Those are special cases though: the general case is that clients who miss the reconnect window must be fenced. The default reconnect timeout in the MDS is rather low (60s iirc), and we may want to revise that upwards when putting a more aggressive client eviction in place.

There is also the question of how a client should respond to being blacklisted: to help users out, we might want a mechanism for client mounts to discard their existing global ID and acquire a new one when they are blacklisted, and then continue to operate normally after cancelling all buffers (effectively as if they had been remounted).