Actions
Bug #7212
closedmonitor fails to start
% Done:
0%
Source:
Q/A
Tags:
Backport:
emperor, dumpling
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Some teuthology/ceph-deploy tests are failing for the ceph master branch because some monitors are
not coming up:
2014-01-23T03:56:45.630 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.vpm077 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] === mon.vpm077 === 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting Ceph mon.vpm077 on vpm077... 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting ceph-create-keys on vpm077... 2014-01-23T03:56:52.905 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;33mWARNIN[0m] No data was received after 7 seconds, disconnecting... 2014-01-23T03:56:54.909 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.vpm077.asok mon_status 2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ******************************************************************************** 2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] status for monitor: mon.vpm077 2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] { 2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "election_epoch": 1, 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "extra_probe_peers": [ 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "10.214.138.87:6789/0", 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "10.214.138.112:6789/0" 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ], 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "monmap": { 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "created": "0.000000", 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "epoch": 1, 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "fsid": "ee15ef3b-5f6e-4cc1-bc0e-96436de15789", 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "modified": "0.000000", 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "mons": [ 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] { 2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "addr": "10.214.138.87:6789/0", 2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "name": "vpm020", 2014-01-23T03:56:55.183 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "rank": 0 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] }, 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] { 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "addr": "10.214.138.112:6789/0", 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "name": "vpm046", 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "rank": 1 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] }, 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] { 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "addr": "0.0.0.0:0/2", 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "name": "vpm077", 2014-01-23T03:56:55.187 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "rank": 2 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] } 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ] 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] }, 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "name": "vpm077", 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "outside_quorum": [], 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "quorum": [], 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "rank": -1, 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "state": "electing", 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] "sync_provider": [] 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] } 2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ******************************************************************************** 2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;37mINFO[0m ] monitor: mon.vpm077 is not running
This causes health errors further along the test:
2014-01-23T03:58:47.030 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 2014-01-23T03:58:47.341 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046 2014-01-23T03:58:57.341 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 2014-01-23T03:58:57.626 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046 2014-01-23T03:59:07.626 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 2014-01-23T03:59:07.926 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046
Actions