Bug #1831
closed
mon: should not accept (and should disconnect) session when not in quorum
Added by Greg Farnum over 12 years ago.
Updated over 12 years ago.
Description
This happened on Benjamin. The OSDs ought to be failing the connection and going to a new monitor, but they failed to do so. My first thoughts were that it looks to be waiting through the default 15-minute connection timeout instead of the 3-second interval Sage says it ought to be bailing out after.
(One possibility: it was a networking issue, so probably the TCP connection never sent an EOF or anything; maybe the shorter timeout isn't looking at a broad enough selection of failure modes.)
- Subject changed from OSDs don't connect to a new monitor when theirs is booted out of the cluster to mon: should not accept (and should disconnect) session when not in quorum
I think there are two parts here:
- the mon shouldn't let sessions start if it is not in the quorum. that may actually work already because sessions can't authenticate.
- if the mon drops out of the quorum, it should disconnect sessions after some short-ish time period. (it can't be immediate or every election would trigger expensive reconnects.)
For the disconnect part, we can either drop the connection and let clients hunt for a new monitor, or we can polite tell them to go elsewhere. Since we're out of the quorum, tho, we may not know where to send them, so a disconnect is probably ok for now...
Any other thoughts?
There's two things here, the second being the monitor changes you're focusing on. I need to investigate further why the OSDs stayed connected though — I don't know if their session was working so all was well, or if it disconnected silently due to the networking issues and the OSDs just didn't notice (failing to reconnect elsewhere). Unfortunately networking issues were blocking me from further analysis today — hopefully tomorrow!
Last night a test hit this. The MDS got stuck connected to an out-of-quorum mon, and never stopped being laggy.
- Translation missing: en.field_position set to 33
- Status changed from New to In Progress
A basic stab at this is staggeringly boring — pushed to wip-mon-timeouts. I want to discuss instrumenting Monitor to stay out of quorums (for testing) and implementing internal heartbeats before merging.
- Status changed from In Progress to 4
Updated the branch, it now includes monitor commands and instrumentation so you can drop a monitor out of the quorum. Unfortunately it's hard to get the monitor back in right now, since the monitor quickly goes unreadable in paxos and then it can't authenticate new clients.
- Status changed from 4 to Resolved
Also available in: Atom
PDF