Actions
Bug #14338
closedIncorrect IPs set in /etc/hosts on mira nodes
Added by Yuri Weinstein over 8 years ago. Updated over 8 years ago.
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-deploy
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/
Jobs: 18887, 18889
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-01-08_12:13:02-ceph-deploy-jewel-distro-basic-mira/18887/teuthology.log
2016-01-08T20:45:16.640 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy.mon][WARNING] waiting 20 seconds before retrying 2016-01-08T20:45:36.661 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
and as result:
2016-01-08T20:45:36.891 INFO:teuthology.orchestra.run.mira061.stderr:[ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file:
Updated by Alfredo Deza over 8 years ago
This looks like the monitors can't resolve the mira061 address:
"mons": [
{
"addr": "172.21.8.118:6789/0",
"name": "mira101",
"rank": 0
},
{
"addr": "0.0.0.0:0/1",
"name": "mira061",
"rank": 1
}
]
Basically, this has to work for every host:
nslookup ${HOST}
And it seems that is not working here.
Updated by Yuri Weinstein over 8 years ago
- Project changed from Ceph to sepia
- Status changed from Closed to New
- Assignee set to David Galloway
David, can you take a look at mira061 configuration, pls?
And then close/resolve the ticket.
Updated by Yuri Weinstein over 8 years ago
Also in run: http://pulpito.ceph.com/teuthology-2016-01-13_21:13:01-ceph-deploy-jewel-distro-basic-mira/
JObs: 28704,28706,28707
Updated by Zack Cerza over 8 years ago
(Rewriting my original comment)
I'm a bit confused about this and why it's assigned to David. I don't see what Alfredo was apparently seeing with respect to DNS lookup failures, and I can't reproduce any on mira061, mira101, or mira058...
Updated by David Galloway over 8 years ago
Yeah I'm not so sure this is a testnode config problem. The logs show the hostname is resolved fine but ceph-deploy pulls in the old 10.214 subnet somehow.
2016-01-13T21:18:25.442 INFO:tasks.ceph_deploy:Building ceph cluster using ceph-deploy... 2016-01-13T21:18:25.442 INFO:teuthology.orchestra.run.mira058:Running: 'cd /home/ubuntu/cephtest/ceph-deploy && ./ceph-deploy new mira058.front.sepia.ceph.com mira075.front.sepia.ceph.com' 2016-01-13T21:18:25.639 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ubuntu/.cephdeploy.conf 2016-01-13T21:18:25.641 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] Invoked (1.5.31): ./ceph-deploy new mira058.front.sepia.ceph.com mira075.front.sepia.ceph.com 2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] ceph-deploy options: 2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] username : None 2016-01-13T21:18:25.642 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] func : <function new at 0x7f0daba09500> 2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] verbose : False 2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] overwrite_conf : False 2016-01-13T21:18:25.643 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] quiet : False 2016-01-13T21:18:25.644 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0dab367a28> 2016-01-13T21:18:25.644 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] cluster : ceph 2016-01-13T21:18:25.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] ssh_copykey : True 2016-01-13T21:18:25.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] mon : ['mira058.front.sepia.ceph.com', 'mira075.front.sepia.ceph.com'] 2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] public_network : None 2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] ceph_conf : None 2016-01-13T21:18:25.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] cluster_network : None 2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] default_release : False 2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.cli][INFO ] fsid : None 2016-01-13T21:18:25.648 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Creating new cluster named ceph 2016-01-13T21:18:25.649 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds 2016-01-13T21:18:25.718 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] connection detected need for sudo 2016-01-13T21:18:25.746 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] connected to host: mira058.front.sepia.ceph.com 2016-01-13T21:18:25.747 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] detect platform information from remote host 2016-01-13T21:18:25.766 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] detect machine type 2016-01-13T21:18:25.769 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] find the location of an executable 2016-01-13T21:18:25.772 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][INFO ] Running command: sudo /bin/ip link show 2016-01-13T21:18:25.794 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][INFO ] Running command: sudo /bin/ip addr show 2016-01-13T21:18:25.806 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.6.134'] 2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira058.front.sepia.ceph.com 2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira058 at 10.214.134.134 2016-01-13T21:18:25.808 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds 2016-01-13T21:18:25.823 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connected to host: mira058 2016-01-13T21:18:25.827 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO ] Running command: ssh -CT -o BatchMode=yes mira075.front.sepia.ceph.com 2016-01-13T21:18:26.179 INFO:teuthology.orchestra.run.mira058.stderr:Warning: Permanently added 'mira075.front.sepia.ceph.com,172.21.7.136' (ECDSA) to the list of known hosts. 2016-01-13T21:18:26.369 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connection detected need for sudo 2016-01-13T21:18:26.401 INFO:teuthology.orchestra.run.mira058.stderr:Warning: Permanently added 'mira075.front.sepia.ceph.com,172.21.7.136' (ECDSA) to the list of known hosts. 2016-01-13T21:18:26.581 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] connected to host: mira075.front.sepia.ceph.com 2016-01-13T21:18:26.582 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] detect platform information from remote host 2016-01-13T21:18:26.604 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] detect machine type 2016-01-13T21:18:26.608 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] find the location of an executable 2016-01-13T21:18:26.613 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO ] Running command: sudo /bin/ip link show 2016-01-13T21:18:26.634 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][INFO ] Running command: sudo /bin/ip addr show 2016-01-13T21:18:26.645 INFO:teuthology.orchestra.run.mira058.stderr:[mira075.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.7.136'] 2016-01-13T21:18:26.646 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira075.front.sepia.ceph.com 2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira075 at 172.21.7.136 2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor initial members are ['mira058', 'mira075'] 2016-01-13T21:18:26.647 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.214.134.134', '172.21.7.136']
This part in particular doesn't seem right
2016-01-13T21:18:25.806 INFO:teuthology.orchestra.run.mira058.stderr:[mira058.front.sepia.ceph.com][DEBUG ] IP addresses found: ['172.21.6.134'] 2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Resolving host mira058.front.sepia.ceph.com 2016-01-13T21:18:25.807 INFO:teuthology.orchestra.run.mira058.stderr:[ceph_deploy.new][DEBUG ] Monitor mira058 at 10.214.134.134
Updated by David Galloway over 8 years ago
sigh
$ cat /etc/hosts 127.0.0.1 localhost 10.214.134.128 mira061.front.sepia.ceph.com mira061
I'll work on getting these entries cleaned up
Updated by David Galloway over 8 years ago
- Subject changed from "Some monitors have still not reached quorum" in ceph-deploy-jewel-distro-basic-mira to Incorrect IPs set in /etc/hosts on mira nodes
- Status changed from New to In Progress
Updated by David Galloway over 8 years ago
- Status changed from In Progress to Resolved
Replaced /etc/hosts entries on miras with proper subnet (172.21. vs 10.214) using modified ansible task (static_ip.yml)
Actions