Project

General

Profile

Actions
Copy link

Bug #37351

closed

luminous: 'ceph-create-keys:ceph-mon is not in quorum' exceeds retries on ovh nodes

Added by Brad Hubbard over 5 years ago. Updated over 5 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Brad Hubbard
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v12.2.9
ceph-qa-suite:
ceph-ansible
Crash signature (v1):
Crash signature (v2):

Description

2018-11-20T07:10:48.834 INFO:teuthology.orchestra.run.ovh010.stdout:TASK [ceph-mon : collect admin and bootstrap keys] *****************************
2018-11-20T07:10:48.834 INFO:teuthology.orchestra.run.ovh010.stdout:task path: /home/ubuntu/ceph-ansible/roles/ceph-mon/tasks/ceph_keys.yml:2
2018-11-20T07:10:48.835 INFO:teuthology.orchestra.run.ovh010.stdout:Tuesday 20 November 2018  07:10:48 +0000 (0:00:00.106)       0:05:17.753 ******
2018-11-20T07:11:23.646 INFO:teuthology.orchestra.run.ovh010.stdout:fatal: [ovh086.front.sepia.ceph.com]: FAILED! => {
2018-11-20T07:11:23.646 INFO:teuthology.orchestra.run.ovh010.stdout:    "changed": false,
2018-11-20T07:11:23.646 INFO:teuthology.orchestra.run.ovh010.stdout:    "cmd": [
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "ceph-create-keys",
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "--cluster",
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "ceph",
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "-i",
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "ovh086",
2018-11-20T07:11:23.647 INFO:teuthology.orchestra.run.ovh010.stdout:        "-t",
2018-11-20T07:11:23.648 INFO:teuthology.orchestra.run.ovh010.stdout:        "30" 
2018-11-20T07:11:23.648 INFO:teuthology.orchestra.run.ovh010.stdout:    ],
2018-11-20T07:11:23.648 INFO:teuthology.orchestra.run.ovh010.stdout:    "delta": "0:00:34.458162",
2018-11-20T07:11:23.648 INFO:teuthology.orchestra.run.ovh010.stdout:    "end": "2018-11-20 07:11:23.596210",
2018-11-20T07:11:23.648 INFO:teuthology.orchestra.run.ovh010.stdout:    "rc": 1,
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:    "start": "2018-11-20 07:10:49.138048" 
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:}
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:STDERR:
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.649 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.650 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.651 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.651 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.651 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.651 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.651 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.652 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.652 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.652 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.652 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.652 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.653 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.653 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.653 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.653 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.654 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.654 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.654 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.654 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.655 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.655 INFO:teuthology.orchestra.run.ovh010.stdout:INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
2018-11-20T07:11:23.655 INFO:teuthology.orchestra.run.ovh010.stdout:ceph-mon was not able to join quorum within 0 seconds
2018-11-20T07:11:23.655 INFO:teuthology.orchestra.run.ovh010.stdout:
2018-11-20T07:11:23.655 INFO:teuthology.orchestra.run.ovh010.stdout:
2018-11-20T07:11:23.656 INFO:teuthology.orchestra.run.ovh010.stdout:MSG:
2018-11-20T07:11:23.656 INFO:teuthology.orchestra.run.ovh010.stdout:
2018-11-20T07:11:23.656 INFO:teuthology.orchestra.run.ovh010.stdout:non-zero return code

http://qa-proxy.ceph.com/teuthology/bhubbard-2018-11-20_06:07:02-ceph-ansible-luminous-distro-basic-ovh/3273156/teuthology.log

Actions
Copy link
 #1

Updated by Brad Hubbard over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Brad Hubbard over 5 years ago

  • Project changed from Ceph to RADOS
  • Category deleted (Monitor)
Actions
Copy link
 #4

Updated by Joao Eduardo Luis over 5 years ago

The run doesn't seem to have any logs. It's a bit difficult for me to figure out if the monitors are not in quorum due to the monitors themselves (not being able to talk to each other, or dead), or because they weren't properly deployed.

Anyone with an ansible background available to go through the teuthology.log and figure that out? Alternatively, is there a run with mon logs available?

Actions
Copy link
 #5

Updated by Brad Hubbard over 5 years ago

Joao,

Thanks for looking. Let me see if I can reproduce.

Actions
Copy link
 #7

Updated by Brad Hubbard over 5 years ago

So I ran this manually with 'interactive-on-error: true' and found that one mon was running but on another node we see the following.

$ ip -o -4 a s eth0
2: eth0    inet 158.69.74.7/32 brd 158.69.74.7 scope global dynamic eth0\       valid_lft 77006sec preferred_lft 77006sec

$ sudo cat /var/log/ceph/ceph-mon.ovh039.log
2018-11-21 04:55:31.278450 7f251a4f4000  0 set uid:gid to 167:167 (ceph:ceph)
2018-11-21 04:55:31.278659 7f251a4f4000 -1 unable to find any IP address in networks '158.69.65.196/32' interfaces ''

$ grep network /etc/ceph/ceph.conf
cluster network = 158.69.65.196/32
public network = 158.69.65.196/32

So it looks to me like the address of the NIC on this machine is not in the cluster or public network?

Actions
Copy link
 #8

Updated by Brad Hubbard over 5 years ago

  • Project changed from RADOS to ovh
  • Status changed from New to Duplicate
Actions
Copy link

Also available in: Atom PDF