mon: should not accept (and should disconnect) session when not in quorum
This happened on Benjamin. The OSDs ought to be failing the connection and going to a new monitor, but they failed to do so. My first thoughts were that it looks to be waiting through the default 15-minute connection timeout instead of the 3-second interval Sage says it ought to be bailing out after.
(One possibility: it was a networking issue, so probably the TCP connection never sent an EOF or anything; maybe the shorter timeout isn't looking at a broad enough selection of failure modes.)
#1 Updated by Sage Weil over 8 years ago
- Subject changed from OSDs don't connect to a new monitor when theirs is booted out of the cluster to mon: should not accept (and should disconnect) session when not in quorum
I think there are two parts here:
- the mon shouldn't let sessions start if it is not in the quorum. that may actually work already because sessions can't authenticate.
- if the mon drops out of the quorum, it should disconnect sessions after some short-ish time period. (it can't be immediate or every election would trigger expensive reconnects.)
For the disconnect part, we can either drop the connection and let clients hunt for a new monitor, or we can polite tell them to go elsewhere. Since we're out of the quorum, tho, we may not know where to send them, so a disconnect is probably ok for now...
Any other thoughts?
#2 Updated by Greg Farnum over 8 years ago
There's two things here, the second being the monitor changes you're focusing on. I need to investigate further why the OSDs stayed connected though — I don't know if their session was working so all was well, or if it disconnected silently due to the networking issues and the OSDs just didn't notice (failing to reconnect elsewhere). Unfortunately networking issues were blocking me from further analysis today — hopefully tomorrow!
#6 Updated by Greg Farnum over 8 years ago
- Status changed from In Progress to 4
Updated the branch, it now includes monitor commands and instrumentation so you can drop a monitor out of the quorum. Unfortunately it's hard to get the monitor back in right now, since the monitor quickly goes unreadable in paxos and then it can't authenticate new clients.