Project

General

Profile

Actions
Copy link

Bug #14338

closed

Incorrect IPs set in /etc/hosts on mira nodes

Added by Yuri Weinstein over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
David Galloway
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-deploy
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/18887/teuthology.log


2016-01-08T20:45:16.640 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy.mon][WARNING] waiting 20 seconds before retrying
2016-01-08T20:45:36.661 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:

and as result:

2016-01-08T20:45:36.891 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file:
Actions
Copy link
 #1

Updated by Yuri Weinstein over 8 years ago

  • Project changed from sepia to Ceph
Actions
Copy link
 #2

Updated by Alfredo Deza over 8 years ago

This looks like the monitors can't resolve the mira061 address:

"mons": [
  {
    "addr": "172.21.8.118:6789/0",
    "name": "mira101",
    "rank": 0
  },
  {
    "addr": "0.0.0.0:0/1",
    "name": "mira061",
    "rank": 1
  }
]

Basically, this has to work for every host:

nslookup ${HOST}

And it seems that is not working here.

Actions
Copy link
 #3

Updated by Samuel Just over 8 years ago

  • Status changed from New to Closed
Actions
Copy link
 #4

Updated by Yuri Weinstein over 8 years ago

  • Project changed from Ceph to sepia
  • Status changed from Closed to New
  • Assignee set to David Galloway

David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket.

Actions
Copy link
 #6

Updated by Zack Cerza over 8 years ago

(Rewriting my original comment)

I'm a bit confused about this and why it's assigned to David. I don't see what Alfredo was apparently seeing with respect to DNS lookup failures, and I can't reproduce any on mira061, mira101, or mira058...

Actions
Copy link
 #7

Updated by David Galloway over 8 years ago

Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy pulls in the old 10.214 subnet somehow.

2016-01-13T21:18:25.442 INFO:tasks.ceph_deploy:Building ceph cluster using ceph-deploy...
2016-01-13T21:18:25.442 INFO:teuthology.orchestra.run.mira058:Running: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy new mira058.front.sepia.ceph.com mira075.front.sepia.ceph.com'
2016-01-13T21:18:25.639 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf
2016-01-13T21:18:25.641 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ] Invoked (1.5.31): ./ceph-deploy new mira058.front.sepia.ceph.com mira075.front.sepia.ceph.com
2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ] ceph-deploy options:
2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  username                      : None
2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7f0daba09500>
2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  verbose                       : False
2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  quiet                         : False
2016-01-13T21:18:25.644 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0dab367a28>
2016-01-13T21:18:25.644 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  cluster                       : ceph
2016-01-13T21:18:25.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
2016-01-13T21:18:25.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  mon                           : ['mira058.front.sepia.ceph.com', 'mira075.front.sepia.ceph.com']
2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  public_network                : None
2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  cluster_network               : None
2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  default_release               : False
2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO  ]  fsid                          : None
2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
2016-01-13T21:18:25.649 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
2016-01-13T21:18:25.718 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] connection detected need for sudo
2016-01-13T21:18:25.746 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] connected to host: mira058.front.sepia.ceph.com
2016-01-13T21:18:25.747 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2016-01-13T21:18:25.766 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] detect machine type
2016-01-13T21:18:25.769 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] find the location of an executable
2016-01-13T21:18:25.772 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][INFO  ] Running command: sudo /bin/ip link show
2016-01-13T21:18:25.794 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][INFO  ] Running command: sudo /bin/ip addr show
2016-01-13T21:18:25.806 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.6.134']
2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira058.front.sepia.ceph.com
2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira058 at 10.214.134.134
2016-01-13T21:18:25.808 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
2016-01-13T21:18:25.823 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connected to host: mira058
2016-01-13T21:18:25.827 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO  ] Running command: ssh -CT -o BatchMode=yes mira075.front.sepia.ceph.com
2016-01-13T21:18:26.179 INFO:teuthology.orchestra.run.mira058.stderr:Warning: Permanently added 'mira075.front.sepia.ceph.com,172.21.7.136' (ECDSA) to the list of known hosts.
2016-01-13T21:18:26.369 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connection detected need for sudo
2016-01-13T21:18:26.401 INFO:teuthology.orchestra.run.mira058.stderr:Warning: Permanently added 'mira075.front.sepia.ceph.com,172.21.7.136' (ECDSA) to the list of known hosts.
2016-01-13T21:18:26.581 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connected to host: mira075.front.sepia.ceph.com
2016-01-13T21:18:26.582 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] detect platform information from remote host
2016-01-13T21:18:26.604 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] detect machine type
2016-01-13T21:18:26.608 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] find the location of an executable
2016-01-13T21:18:26.613 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO  ] Running command: sudo /bin/ip link show
2016-01-13T21:18:26.634 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO  ] Running command: sudo /bin/ip addr show
2016-01-13T21:18:26.645 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.7.136']
2016-01-13T21:18:26.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira075.front.sepia.ceph.com
2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira075 at 172.21.7.136
2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor initial members are ['mira058', 'mira075']
2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.134.134', '172.21.7.136']

This part in particular doesn't seem right

2016-01-13T21:18:25.806 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.6.134']
2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira058.front.sepia.ceph.com
2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira058 at 10.214.134.134

Actions
Copy link
 #8

Updated by David Galloway over 8 years ago

sigh

$ cat /etc/hosts
127.0.0.1    localhost
10.214.134.128 mira061.front.sepia.ceph.com mira061

I'll work on getting these entries cleaned up

Actions
Copy link
 #9

Updated by David Galloway over 8 years ago

  • Subject changed from "Some monitors have still not reached quorum" in ceph-deploy-jewel-distro-basic-mira to Incorrect IPs set in /etc/hosts on mira nodes
  • Status changed from New to In Progress
Actions
Copy link
 #10

Updated by David Galloway over 8 years ago

  • Status changed from In Progress to Resolved

Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml)

Actions
Copy link

Also available in: Atom PDF