Bug #1831
closedmon: should not accept (and should disconnect) session when not in quorum
0%
Description
This happened on Benjamin. The OSDs ought to be failing the connection and going to a new monitor, but they failed to do so. My first thoughts were that it looks to be waiting through the default 15-minute connection timeout instead of the 3-second interval Sage says it ought to be bailing out after.
(One possibility: it was a networking issue, so probably the TCP connection never sent an EOF or anything; maybe the shorter timeout isn't looking at a broad enough selection of failure modes.)
Updated by Sage Weil over 12 years ago
- Subject changed from OSDs don't connect to a new monitor when theirs is booted out of the cluster to mon: should not accept (and should disconnect) session when not in quorum
I think there are two parts here:
- the mon shouldn't let sessions start if it is not in the quorum. that may actually work already because sessions can't authenticate.
- if the mon drops out of the quorum, it should disconnect sessions after some short-ish time period. (it can't be immediate or every election would trigger expensive reconnects.)For the disconnect part, we can either drop the connection and let clients hunt for a new monitor, or we can polite tell them to go elsewhere. Since we're out of the quorum, tho, we may not know where to send them, so a disconnect is probably ok for now...
Any other thoughts?
Updated by Greg Farnum over 12 years ago
There's two things here, the second being the monitor changes you're focusing on. I need to investigate further why the OSDs stayed connected though — I don't know if their session was working so all was well, or if it disconnected silently due to the networking issues and the OSDs just didn't notice (failing to reconnect elsewhere). Unfortunately networking issues were blocking me from further analysis today — hopefully tomorrow!
Updated by Josh Durgin over 12 years ago
Last night a test hit this. The MDS got stuck connected to an out-of-quorum mon, and never stopped being laggy.
Updated by Sage Weil over 12 years ago
- Translation missing: en.field_position set to 33
Updated by Greg Farnum over 12 years ago
- Status changed from New to In Progress
A basic stab at this is staggeringly boring — pushed to wip-mon-timeouts. I want to discuss instrumenting Monitor to stay out of quorums (for testing) and implementing internal heartbeats before merging.
Updated by Greg Farnum over 12 years ago
- Status changed from In Progress to 4
Updated the branch, it now includes monitor commands and instrumentation so you can drop a monitor out of the quorum. Unfortunately it's hard to get the monitor back in right now, since the monitor quickly goes unreadable in paxos and then it can't authenticate new clients.
Updated by Sage Weil over 12 years ago
- Status changed from 4 to Resolved