Ceph » CephFS

Okay, I believe the important sequence is:

• client creates inode, mds replies unsafe
• client requests inode change gid 65533, mds replies unsafe
• client requests inode change gid 65532, mds crashes
• mds goes into replay
• client sends off caps, which include inode on gid 65533
• mds does not recognize inode number
• mds drops client cap update on clientreplay_start, because it can't find inode
• client replays create and two setattrs
• client does not update trace on replies because it has already has same seq (1)
• client sends cap update, which still has gid 65533, mds accepts it

So I'm not quite sure what to do here. We have tids set by the client, and we have seqs set by the server, but seqs get reset when the MDS does and cap tids aren't related to regular op tids. I think for any missing inodes we ought to be trying to apply the caps after the messages get replayed, rather than just dropping them, but that actually doesn't help us here because we'd be going backwards anyway.

...actually, I don't see what keeps a client cap flush from overwriting a setattr change under non-failure cases, if for some reason the response manages to take long enough for caps to get flushed in the middle.

Okay, this is running into code from the uid/gid enforcement stuff. From https://github.com/ceph/ceph/commit/1957aeddbf05f2ecf3be0a760ff5b5c313370eea in particular, which is justified by the comment in https://github.com/ceph/ceph/commit/ac031443 — but I don't think that's a true statement. Many of our other update functions invoke check_caps, but _setattr doesn't and even if it did (prior to making changes) there's no reason to think the dirtied cap bits would force an immediate flush. So I think we need to force a flush if we're doing a sync setattr while we have dirty caps.

But that's not a complete fix. We still need a way of ordering cap updates with respect to requested setattr operations.

I interpret this differently.

• client creates inode, mds replies unsafe
• client requests inode change gid 65533, client marks Ax dirty
• client requests inode change gid 65532, client sends setattr request to MDS
• mds crashes and goes into reconnect stage
• client re-sends the setattr request (client does not got unsafe, see Client::resend_unsafe_requests)
• mds decide not to handle the setattr request in client_replay stage. (see Server::dispatch)
• client re-sends the cap flush with gid 65533 (see Client::early_kick_flushing_caps)
• mds goes to client_replay stags
• mds drops client cap update on clientreplay_start, because it can't find inode
• mds replays the create
• mds goto to active
• mds handles the setattr request and send replay to client (
• client re-sends the cap flush with gid 65533 again (see Client::kick_flushing_caps)
• client does not update trace on replies because it has already has same seq (1)
• client sends cap update, which still has gid 65533, mds accepts it