Project

General

Profile

Actions
Copy link

Bug #53033

open

cephadm removes MONs during upgrade 15.2.14 > 16.2.6 which leads to failed quorum and broken cluster

Added by Tobias Fischer over 2 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
Ceph - v15.2.14, Ceph - v16.2.4
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I started Upgrading our Production Clusters from Octopus v15.2.14 to Pacific v16.2.6 via Cephadm Orchestrator. The Upgrade of the first 4 Clusters went fine. Today I started the Upgrade of our biggest Cluster ( 1k OSDs / 3 PiB ) which lead to a broken Cluster during Upgrade ( no MON quorum ).
After some Analysis we found out that Cephadm decided after the 1st (out of 5) MON was running Pacific to remove 2 of the other MONs:

Oct 25 09:01:08 cephadm [INF] Filtered out host pod2-sc2: does not belong to mon public_network (10.27.251.128/25)
Oct 25 09:01:08 cephadm [INF] Filtered out host pod2-sc3: does not belong to mon public_network (10.27.251.128/25)
Oct 25 09:01:08 cephadm [INF] Safe to remove mon.pod2-sc1: new quorum should be ['pod2-mon5', 'pod2-mon6', 'pod2-sc2', 'pod2-sc3'] (from ['pod2-mon5', 'pod2-mon6', 'pod2-sc2', 'pod2-sc3'])
Oct 25 09:01:08 cephadm [INF] Removing monitor pod2-sc1 from monmap...
Oct 25 09:01:14 cephadm [INF] Safe to remove mon.pod2-sc2: new quorum should be ['pod2-mon5', 'pod2-mon6', 'pod2-sc3'] (from ['pod2-mon5', 'pod2-mon6', 'pod2-sc3'])
Oct 25 09:01:14 cephadm [INF] Removing monitor pod2-sc2 from monmap...

The difference of this Cluster compared to the successfully upgraded ones: The others had MONs in umanaged state - this one had MONs managed.
To Fix the Issue we stopped the upgrade, disabled cephadm and stopped the upgraded MON via docker stop, wait for other MONs to build quorum and Cluster recovery, resume cephadm, set the MONs to unmanaged and restarted Upgrade.

To prevent such dangerous situations I would propose to implement A stop of all Cephadm/Orchestartor changes while a upgrade is running. It really doesn't make sense to remove Monitors/do any Changes while a upgrade is running

Related issues 1 (0 open — 1 closed)

Related to Orchestrator - Bug #51027: monmap drops rebooted mon if deployed via labelResolvedAdam King

Actions
Actions
Copy link
 #1

Updated by Tobias Fischer over 2 years ago

monmaps during upgrade

epoch 33
fsid 5cd67aaf-e0e8-4822-8d38-d5b0214a9a9b
last_changed 2020-11-02T12:38:02.172399+0000
created 2016-08-02T09:47:28.351216+0000
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.27.251.134:3300/0,v1:10.27.251.134:6789/0] mon.pod2-mon5
1: [v2:10.27.251.135:3300/0,v1:10.27.251.135:6789/0] mon.pod2-mon6
2: [v2:10.27.251.247:3300/0,v1:10.27.251.247:6789/0] mon.pod2-sc1
3: [v2:10.27.251.250:3300/0,v1:10.27.251.250:6789/0] mon.pod2-sc2
4: [v2:10.27.251.251:3300/0,v1:10.27.251.251:6789/0] mon.pod2-sc3


epoch 34
fsid 5cd67aaf-e0e8-4822-8d38-d5b0214a9a9b
last_changed 2021-10-25T07:01:00.277500+0000
created 2016-08-02T09:47:28.351216+0000
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.27.251.134:3300/0,v1:10.27.251.134:6789/0] mon.pod2-mon5
1: [v2:10.27.251.135:3300/0,v1:10.27.251.135:6789/0] mon.pod2-mon6
2: [v2:10.27.251.250:3300/0,v1:10.27.251.250:6789/0] mon.pod2-sc2
3: [v2:10.27.251.251:3300/0,v1:10.27.251.251:6789/0] mon.pod2-sc3


epoch 35
fsid 5cd67aaf-e0e8-4822-8d38-d5b0214a9a9b
last_changed 2021-10-25T07:01:11.707327+0000
created 2016-08-02T09:47:28.351216+0000
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.27.251.134:3300/0,v1:10.27.251.134:6789/0] mon.pod2-mon5
1: [v2:10.27.251.135:3300/0,v1:10.27.251.135:6789/0] mon.pod2-mon6
2: [v2:10.27.251.251:3300/0,v1:10.27.251.251:6789/0] mon.pod2-sc3


epoch 36
fsid 5cd67aaf-e0e8-4822-8d38-d5b0214a9a9b
last_changed 2021-10-25T07:04:25.698273+0000
created 2016-08-02T09:47:28.351216+0000
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.27.251.134:3300/0,v1:10.27.251.134:6789/0] mon.pod2-mon5
1: [v2:10.27.251.135:3300/0,v1:10.27.251.135:6789/0] mon.pod2-mon6
2: [v2:10.27.251.251:3300/0,v1:10.27.251.251:6789/0] mon.pod2-sc3
3: [v2:10.27.251.247:3300/0,v1:10.27.251.247:6789/0] mon.pod2-sc1


epoch 37
fsid 5cd67aaf-e0e8-4822-8d38-d5b0214a9a9b
last_changed 2021-10-25T07:31:44.991971+0000
created 2016-08-02T09:47:28.351216+0000
min_mon_release 15 (octopus)
election_strategy: 1
0: [v2:10.27.251.134:3300/0,v1:10.27.251.134:6789/0] mon.pod2-mon5
1: [v2:10.27.251.135:3300/0,v1:10.27.251.135:6789/0] mon.pod2-mon6
2: [v2:10.27.251.251:3300/0,v1:10.27.251.251:6789/0] mon.pod2-sc3
3: [v2:10.27.251.247:3300/0,v1:10.27.251.247:6789/0] mon.pod2-sc1
4: [v2:10.27.251.250:3300/0,v1:10.27.251.250:6789/0] mon.pod2-sc2

Actions
Copy link
 #2

Updated by Sebastian Wagner over 2 years ago

  • Related to Bug #51027: monmap drops rebooted mon if deployed via label added
Actions
Copy link
 #3

Updated by Sebastian Wagner over 2 years ago

Tobias, is this a duplicate of #51027 ?

Actions
Copy link
 #4

Updated by Tobias Fischer over 2 years ago

I don't think it's the same bug. In the other bug the mon was removed from the monmap (by orchestrator?) after a reboot. In this Bug the mon got removed by orchestrator during upgrade after 1st mon got updated. But maybe the bugs are related. the best idea would probably be to try to repeate this with the patch applied from the other bug.

Actions
Copy link

Also available in: Atom PDF