Ceph » Linux kernel client

I believe this is caused by "ceph: drop the messages from MDS when unmounting" in the testing branch.

In kclient we need to increase the sessions' seq just before dropping the `cap grant` message.

Patrick,

I am thinking whether should we fix the code in case when kclient receives a corrupted msg, which will also skip increasing and resending the close request too ? IMO even the msg corrupted but the MDS side should have increased the seq already and in client side we should increase it anyway.

This is unrelated to the ceph: drop the messages from MDS when unmounting commit, this is an issue existing long time ago.

Xiubo Li wrote:

Patrick,

I am thinking whether should we fix the code in case when kclient receives a corrupt msg, which will also skip increasing and resending the close request too ?

Like a legit corrupt message that cannot be decoded? If a message cannot be decoded, we routinely drop it in Ceph. I would expect the kernel should have some sort of "stop all i/o and render the mount unusable" if that happens. I don't think that's necessarily relevant for this ticket though.

IMO even the msg corrupted but the MDS side should have increased the seq already and in client side we should increase it anyway.

The MDS did increase the seq and send a new cap message. The problem is the kernel didn't resend the close request after seeing the new seq bump. Even more, the kernel completed unmount without waiting for the CEPH_SESSION_CLOSE message from the mds (no --force flag was used!).

This is unrelated to the ceph: drop the messages from MDS when unmounting commit, this is a issue existing long time ago.

Okay, I'm surprised we haven't seen this before then...

Patrick Donnelly wrote:

Xiubo Li wrote:

Patrick,

I am thinking whether should we fix the code in case when kclient receives a corrupt msg, which will also skip increasing and resending the close request too ?

Like a legit corrupt message that cannot be decoded? If a message cannot be decoded, we routinely drop it in Ceph. I would expect the kernel should have some sort of "stop all i/o and render the mount unusable" if that happens. I don't think that's necessarily relevant for this ticket though.

Yeah, kclient will drop the corrupted messages in some case while in some cases we will mark the whole mounter readonly. It will depends.

IMO even the msg corrupted but the MDS side should have increased the seq already and in client side we should increase it anyway.

The MDS did increase the seq and send a new cap message. The problem is the kernel didn't resend the close request after seeing the new seq bump. Even more, the kernel completed unmount without waiting for the CEPH_SESSION_CLOSE message from the mds (no --force flag was used!).

No, the kclient will wait for CEPH_SESSION_CLOSE messages from MDSs for 60 seconds by default. After that it will tear down the sessions.

The ceph: drop the messages from MDS when unmounting commit is trying to fix one reference leakage bug, which could cause use-after-free crash for fs/crypto/, for fscrypt feature.

And when unmounting after all the in-flight IOs have finished, which all reqs have gotten the replies from MDSs/OSDs, the MDS may continue sending new cap messages, which make no sense anymore, to kclient. This commit is trying to drop them. But forgot to resend the close session request before dropping them. I have fixed it already in [1].

This is unrelated to the ceph: drop the messages from MDS when unmounting commit, this is a issue existing long time ago.

Okay, I'm surprised we haven't seen this before then...

Because it's a corner case when unmounting the kclient will it receive a corrupted msg. Months ago Venky and I hit one issue that the MDS sent one msg to kclient and the kclient couldn't correctly decode it due to the version mismatch issue. And the kclient dropped that msg directly.

In this case the MDS side has increased the seq anyway.

[1] https://patchwork.kernel.org/project/ceph-devel/list/?series=734003