Bug #59688
openmds: idempotence issue in client request
0%
Description
Found the mds may process a same client request twice after session with client rebuild because the network issue.
Case
If a client request has been processed on the MDS side but the client did not receive any reply from the MDS due to network issues, and the session is closed due to idle, see what happens when the session reconnects, notice the client request will be resend with the same tid and +1 retry attempt.
We test this case with a unlink request.
Conclusion
MDS Cannot handle this case and it'll return -2 (No such file or directory) to the Client, even if the file existed before the request was sent, which may exceed the expectations of the user application.
The discussion with @Venky here: https://github.com/ceph/ceph/pull/46050#issuecomment-1508016314
Configuration
client_reconnect_stale = true
mds_session_blocklist_on_evict = false
mds_session_blocklist_on_timeout = false
Step to reproduce
1. Set the breakpoint in Server::journal_and_reply, and continue
2. Client send a client request (unlink) to MDS to trigger the breakpoint
3. On Client, block INPUT/OUTPUT messages from MDS (using iptables, run `iptables -A INPUT/OUTPUT -p tcp --dport 6826:6827 -j DROP`)
4. Continue MDS, now client will drop the unsafe reply from MDS, and can't send the renewcaps to MDS
5. Wait for session timed out (about 60s/300s, depends on find_idle_sessions), MDS will kill the session with Client (will Client session stale immediately? no)
6. Delete the iptables rules, Client will try to reconnect the (stale?) session with MDS
7. Client will resend the client request (unlink) to MDS with the same tid and +1 retry attempt
Log
# first request to mds
2023-05-08T16:35:52.398+0800 7fd2827fc640 10 client.4374 send_request client_request(unknown.0:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 caller_uid=0, caller_gid=0{0,1001,}) v6 to mds.0
# mds
2023-05-08T16:35:52.398+0800 7fd16a7fc640 4 mds.0.server handle_client_request client_request(client.4374:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 caller_uid=0, caller_gid=0{0,1001,}) v6
...
# unsafe_reply, we block here and set the iptables rules, so client did not received that
2023-05-08T16:36:05.858+0800 7fd16a7fc640 10 mds.0.server early_reply 0 ((0) Success) client_request(client.4374:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 caller_uid=0, caller_gid=0{0,1001,}) v6
...
2023-05-08T16:36:05.868+0800 7fd13b7fe640 7 mds.0.cache request_finish request(client.4374:19 nref=2 cr=0x7fd164049050)
...
# kill_session when timed out
2023-05-08T16:40:55.878+0800 7fd168ff9640 10 mds.0.server autoclosing stale session client.4374 172.20.214.138:0/405114626 last renewed caps 316.63s ago
2023-05-08T16:40:55.878+0800 7fd168ff9640 10 mds.0.server kill_session 0x7fd1540381a0
# client, rebuild session after we deleted the iptables rules
2023-05-08T16:41:43.345+0800 7fd2a57fa640 0 client.4374 ms_handle_remote_reset on v2:172.20.214.138:6826/3512588839
2023-05-08T16:41:43.345+0800 7fd2a57fa640 1 client.4374 reset from mds we were open; close mds session for reconnect
2023-05-08T16:41:43.345+0800 7fd2a57fa640 10 client.4374 kick_requests_closed for mds.0
...
# client request will waiting for session reconnect because is not got_unsafe
2023-05-08T16:41:43.345+0800 7fd280ff9640 10 client.4374 waiting for session to mds.0 to open
...
# session rebuilt
2023-05-08T16:41:43.585+0800 7fd2a57fa640 10 client.4374 handle_client_session client_session(open) v5 from mds.0
...
# resend client request, retry=1
2023-05-08T16:41:43.585+0800 7fd2827fc640 10 client.4374 send_request rebuilding request 19 for mds.0
2023-05-08T16:41:43.585+0800 7fd2827fc640 10 client.4374 send_request client_request(unknown.0:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 RETRY=1 caller_uid=0, caller_gid=0{0,1001,}) v6 to mds.0
# mds process this client request again, but it is already deleted, return no such file
2023-05-08T16:41:43.585+0800 7fd16a7fc640 4 mds.0.server handle_client_request client_request(client.4374:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 RETRY=1 caller_uid=0, caller_gid=0{0,1001,}) v6
...
2023-05-08T16:41:43.585+0800 7fd16a7fc640 7 mds.0.server reply_client_request -2 ((2) No such file or directory) client_request(client.4374:19 unlink #0x1/222 2023-05-08T16:35:52.404740+0800 RETRY=1 caller_uid=0, caller_gid=0{0,1001,}) v6
# client got unexpected result
2023-05-08T16:41:43.585+0800 7fd2827fc640 8 client.4374 unlink(#0x1/222) = -2
Updated by Venky Shankar 12 months ago
- Status changed from New to Triaged
- Backport set to reef,quincy,pacific
Hi Mer Xuanyi,
Thanks for the detailed report.
So, this requires flipping the config tunables from their defaults as you describe. Although the outcome is not ideal (unlink returning -ENOENT), I wonder if something that is expected especially when enabling `client_reconnect_stale` and blocklisting being disabled.
I did a quick grep in the ceph source for `mds_session_blocklist` under qa tests and nothing came up. So, there aren't any tests that use such configurations which makes me think again if such behavior from the MDS is expected.
You did find a bug however, but we need to assess if its worthwhile fixing it.
Updated by Mer Xuanyi 12 months ago
Venky Shankar wrote:
Hi Mer Xuanyi,
Thanks for the detailed report.
So, this requires flipping the config tunables from their defaults as you describe. Although the outcome is not ideal (unlink returning -ENOENT), I wonder if something that is expected especially when enabling `client_reconnect_stale` and blocklisting being disabled.
I did a quick grep in the ceph source for `mds_session_blocklist` under qa tests and nothing came up. So, there aren't any tests that use such configurations which makes me think again if such behavior from the MDS is expected.
You did find a bug however, but we need to assess if its worthwhile fixing it.
In my understanding, if we set `mds_session_block_xxx` to false and `client_reconnect_stale` to true, we expect the client is able to auto repair from a network breakdown, which is common in distributed systems.
It should be imperceptible to the user of file system, and we won't need to unblock these clients manually.
Updated by Venky Shankar 12 months ago
Mer Xuanyi wrote:
Venky Shankar wrote:
Hi Mer Xuanyi,
Thanks for the detailed report.
So, this requires flipping the config tunables from their defaults as you describe. Although the outcome is not ideal (unlink returning -ENOENT), I wonder if something that is expected especially when enabling `client_reconnect_stale` and blocklisting being disabled.
I did a quick grep in the ceph source for `mds_session_blocklist` under qa tests and nothing came up. So, there aren't any tests that use such configurations which makes me think again if such behavior from the MDS is expected.
You did find a bug however, but we need to assess if its worthwhile fixing it.
In my understanding, if we set `mds_session_block_xxx` to false and `client_reconnect_stale` to true, we expect the client is able to auto repair from a network breakdown, which is common in distributed systems.
It should be imperceptible to the user of file system, and we won't need to unblock these clients manually.
You have a valid point. The lack of tests with these options being non-default are making me nervous that we might uncover other bugs when put to test. I'll check with Patrick and Greg who have more history regarding these config options than me.
But, yes, we need to start this by adding tests first.