Project

General

Profile

Actions

Bug #58049

closed

mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorum

Added by Kamoltat (Junior) Sirivadhna over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Urgent
Category:
Stretch Clusters
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

First encountered in the downstream: https://bugzilla.redhat.com/show_bug.cgi?id=2142674

When we failover monitors many times in the stretch cluster, there are instances where Ceph becomes
unresponsive due to monitors not being able to form a quorum.

We have investigated and concluded that this is due to how we mishandled `removed_ranks` in MonMap which
leads to inconsistent peer_tracker which then leads to deadlock election state of the monitor, which means
they cannot form a quorum -> ceph becomes unresponsive.


Related issues 3 (0 open3 closed)

Related to RADOS - Bug #58107: mon-stretch: old stretch_marked_down_mons leads to ceph unresponsiveClosedKamoltat (Junior) Sirivadhna

Actions
Copied to RADOS - Backport #58380: pacific: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorumResolvedKamoltat (Junior) SirivadhnaActions
Copied to RADOS - Backport #58381: quincy: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorumResolvedKamoltat (Junior) SirivadhnaActions
Actions #1

Updated by Radoslaw Zarzynski over 1 year ago

  • Pull request ID set to 48991
Actions #2

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

  • Related to Bug #58107: mon-stretch: old stretch_marked_down_mons leads to ceph unresponsive added
Actions #3

Updated by Radoslaw Zarzynski over 1 year ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific,quincy
Actions #4

Updated by Backport Bot over 1 year ago

  • Copied to Backport #58380: pacific: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorum added
Actions #5

Updated by Backport Bot over 1 year ago

  • Copied to Backport #58381: quincy: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker leading to unable to form quorum added
Actions #6

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions #7

Updated by Kamoltat (Junior) Sirivadhna about 1 year ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF