Bug #7212: monitor fails to start - Ceph - Ceph

Bug #7212

Updated by Alfredo Deza over 10 years ago

Some teuthology/ceph-deploy tests are failing for the ceph master branch because some monitors are 
 not coming up: 

 <pre> 
 2014-01-23T03:56:45.630 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m    ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.vpm077 
 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] === mon.vpm077 === 
 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting Ceph mon.vpm077 on vpm077... 
 2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting ceph-create-keys on vpm077... 
 2014-01-23T03:56:52.905 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;33mWARNIN[0m] No data was received after 7 seconds, disconnecting... 
 2014-01-23T03:56:54.909 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m    ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.vpm077.asok mon_status 
 2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ******************************************************************************** 
 2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] status for monitor: mon.vpm077 
 2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] { 
 2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "election_epoch": 1, 
 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "extra_probe_peers": [ 
 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "10.214.138.87:6789/0", 
 2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "10.214.138.112:6789/0" 
 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     ], 
 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "monmap": { 
 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "created": "0.000000", 
 2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "epoch": 1, 
 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "fsid": "ee15ef3b-5f6e-4cc1-bc0e-96436de15789", 
 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "modified": "0.000000", 
 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       "mons": [ 
 2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         { 
 2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "addr": "10.214.138.87:6789/0", 
 2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "name": "vpm020", 
 2014-01-23T03:56:55.183 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "rank": 0 
 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         }, 
 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         { 
 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "addr": "10.214.138.112:6789/0", 
 2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "name": "vpm046", 
 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "rank": 1 
 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         }, 
 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         { 
 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "addr": "0.0.0.0:0/2", 
 2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "name": "vpm077", 
 2014-01-23T03:56:55.187 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]           "rank": 2 
 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         } 
 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       ] 
 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     }, 
 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "name": "vpm077", 
 2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "outside_quorum": [], 
 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "quorum": [], 
 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "rank": -1, 
 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "state": "electing", 
 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "sync_provider": [] 
 2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] } 
 2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ******************************************************************************** 
 2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;37mINFO[0m    ] monitor: mon.vpm077 is not running 
 </pre>  

 This causes health errors further along the test: 

 <pre> 
 2014-01-23T03:58:47.030 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 
 2014-01-23T03:58:47.341 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046 
 2014-01-23T03:58:57.341 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 
 2014-01-23T03:58:57.626 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046 
 2014-01-23T03:59:07.626 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health' 
 2014-01-23T03:59:07.926 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046 
 </pre> 

 Log file: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-22_06:21:18-ceph-deploy-master-testing-basic-vps/48108/teuthology.log

Back

Project

General

Profile

Ceph

Bug #7212