Project

General

Profile

Bug #1831

mon: should not accept (and should disconnect) session when not in quorum

Added by Greg Farnum almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Monitor
Target version:
Start date:
12/14/2011
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

This happened on Benjamin. The OSDs ought to be failing the connection and going to a new monitor, but they failed to do so. My first thoughts were that it looks to be waiting through the default 15-minute connection timeout instead of the 3-second interval Sage says it ought to be bailing out after.
(One possibility: it was a networking issue, so probably the TCP connection never sent an EOF or anything; maybe the shorter timeout isn't looking at a broad enough selection of failure modes.)


Related issues

Related to Ceph - Bug #1865: mon: need to disconnect clients when we drop out of quorum Duplicate 12/29/2011

History

#1 Updated by Sage Weil almost 8 years ago

  • Subject changed from OSDs don't connect to a new monitor when theirs is booted out of the cluster to mon: should not accept (and should disconnect) session when not in quorum

I think there are two parts here:

- the mon shouldn't let sessions start if it is not in the quorum.  that may actually work already because sessions can't authenticate.
- if the mon drops out of the quorum, it should disconnect sessions after some short-ish time period. (it can't be immediate or every election would trigger expensive reconnects.)

For the disconnect part, we can either drop the connection and let clients hunt for a new monitor, or we can polite tell them to go elsewhere. Since we're out of the quorum, tho, we may not know where to send them, so a disconnect is probably ok for now...

Any other thoughts?

#2 Updated by Greg Farnum almost 8 years ago

There's two things here, the second being the monitor changes you're focusing on. I need to investigate further why the OSDs stayed connected though — I don't know if their session was working so all was well, or if it disconnected silently due to the networking issues and the OSDs just didn't notice (failing to reconnect elsewhere). Unfortunately networking issues were blocking me from further analysis today — hopefully tomorrow!

#3 Updated by Josh Durgin almost 8 years ago

Last night a test hit this. The MDS got stuck connected to an out-of-quorum mon, and never stopped being laggy.

#4 Updated by Sage Weil almost 8 years ago

  • translation missing: en.field_position set to 33

#5 Updated by Greg Farnum almost 8 years ago

  • Status changed from New to In Progress

A basic stab at this is staggeringly boring — pushed to wip-mon-timeouts. I want to discuss instrumenting Monitor to stay out of quorums (for testing) and implementing internal heartbeats before merging.

#6 Updated by Greg Farnum almost 8 years ago

  • Status changed from In Progress to Feedback

Updated the branch, it now includes monitor commands and instrumentation so you can drop a monitor out of the quorum. Unfortunately it's hard to get the monitor back in right now, since the monitor quickly goes unreadable in paxos and then it can't authenticate new clients.

#7 Updated by Sage Weil almost 8 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF