Project

General

Profile

Bug #38850

upgrade: 1 nautilus mon + 1 luminous mon can't automatically form quorum

Added by Tim Serong 28 days ago. Updated 14 days ago.

Status:
New
Priority:
Normal
Category:
Correctness/Safety
Target version:
-
Start date:
03/22/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:

Description

Seen while upgrading Luminous (12.2.10) to Nautilus (14.2.0). Three mon hosts, four osd hosts. The process was:

- Shutdown mon1 (quorum in now mon2+mon3, both Luminous)
- Upgrade mon1 to Nautilus
- Start mon1 again. mon1 joins cluster, `ceph health` reports all three mons OK
- Shutdown mon2 (leaving mon1 = Nautilus and mon3 = Luminous)
- `ceph health` is now broken (eventually times out)
- mon1 logs repeat:

2019-03-22 09:50:21.634 7f31906d1700  1 mon.mon1@0(electing) e1  peer v1:172.16.1.13:6789/0 release  < min_mon_release, or missing features 0

- mon3 logs repeat:

2019-03-22 09:51:59.644672 7fa9f589a700 -1 mon.mon3@2(probing) e1 handle_probe missing features, have 4611087853745930235, required 0, missing 0

This means that the cluster is effectively down until you're able to complete the upgrade of mon2.

Curiously, on mon1 (Nautilus):

# ceph daemon mon.$(hostname) mon_status|grep min_mon_release
        "min_mon_release": 12,
        "min_mon_release_name": "luminous",

So why is it comlaining about release < min_mon_release?

Even more interesting, I can run this on the Luminous mon:

# ceph daemon mon.$(hostname) quorum enter
started responding to quorum, initiated new election

...and bam a few seconds later, we're in business again:

# ceph status
  cluster:
    id:     44e4a575-5c31-3c61-88c5-001ea49e8aaa
    health: HEALTH_WARN
            1/3 mons down, quorum mon1,mon3

  services:
    mon: 3 daemons, quorum mon1,mon3, out of quorum: mon2
    mgr: mon3(active), standbys: mon1
    osd: 30 osds: 30 up, 30 in

  data:
    pools:   1 pools, 512 pgs
    objects: 0 objects, 0B
    usage:   30.3GiB used, 567GiB / 597GiB avail
    pgs:     512 active+clean

Not that "quorum enter" doesn't help if run from the Nautilus mon, it only works when run from the Luminous mon.

History

#1 Updated by Ricardo Dias 28 days ago

  • Assignee set to Joao Eduardo Luis

#2 Updated by Tim Serong 28 days ago

Just to clarify slightly -- I know the upgrade instructions in the Nautilus release announcement say to "upgrade monitors by installing the new packages and restarting the monitor daemons", but this quick way of upgrading is not always possible; depending on what distro you're using, you may have to upgrade the base OS before you can install the new Nautilus packages, which means that each node is going to be down for quite some time (at least several minutes, maybe many tens of minutes or longer).

#3 Updated by Joao Eduardo Luis 28 days ago

  • Category set to Correctness/Safety
  • Regression changed from No to Yes

#4 Updated by Lars Marowsky-Brée 25 days ago

Agreed, my expectation would be that we can maintain quorum during the entire upgrade period. Even discounting OS upgrades, restarts can go wrong, nodes can fail etc. Maintaining availability is crucial.

#5 Updated by Greg Farnum 14 days ago

This is super weird; the only other recent reference I see to min_mon_release is https://github.com/ceph/ceph/pull/27107 but that looks like just an output bug that we hit when going from Luminous to master/"Octopus" releases...

Also available in: Atom PDF