Project

General

Profile

Actions

Bug #58049

closed

mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorum

Added by Kamoltat (Junior) Sirivadhna over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Urgent
Category:
Stretch Clusters
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

First encountered in the downstream: https://bugzilla.redhat.com/show_bug.cgi?id=2142674

When we failover monitors many times in the stretch cluster, there are instances where Ceph becomes
unresponsive due to monitors not being able to form a quorum.

We have investigated and concluded that this is due to how we mishandled `removed_ranks` in MonMap which
leads to inconsistent peer_tracker which then leads to deadlock election state of the monitor, which means
they cannot form a quorum -> ceph becomes unresponsive.


Related issues 3 (0 open3 closed)

Related to RADOS - Bug #58107: mon-stretch: old stretch_marked_down_mons leads to ceph unresponsiveClosedKamoltat (Junior) Sirivadhna

Actions
Copied to RADOS - Backport #58380: pacific: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorumResolvedKamoltat (Junior) SirivadhnaActions
Copied to RADOS - Backport #58381: quincy: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorumResolvedKamoltat (Junior) SirivadhnaActions
Actions

Also available in: Atom PDF