Project

General

Profile

Actions
Copy link

Bug #7212

closed

monitor fails to start

Added by Alfredo Deza over 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
High
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
emperor, dumpling
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Some teuthology/ceph-deploy tests are failing for the ceph master branch because some monitors are
not coming up:

2014-01-23T03:56:45.630 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m  ] Running command: sudo /sbin/service ceph -c /etc/ceph/ceph.conf start mon.vpm077
2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] === mon.vpm077 ===
2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting Ceph mon.vpm077 on vpm077...
2014-01-23T03:56:45.904 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;34mDEBUG[0m ] Starting ceph-create-keys on vpm077...
2014-01-23T03:56:52.905 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;33mWARNIN[0m] No data was received after 7 seconds, disconnecting...
2014-01-23T03:56:54.909 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077.front.sepia.ceph.com[0m][[1;37mINFO[0m  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.vpm077.asok mon_status
2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ********************************************************************************
2014-01-23T03:56:55.176 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] status for monitor: mon.vpm077
2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] {
2014-01-23T03:56:55.177 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "election_epoch": 1,
2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "extra_probe_peers": [
2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "10.214.138.87:6789/0",
2014-01-23T03:56:55.178 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "10.214.138.112:6789/0" 
2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   ],
2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "monmap": {
2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "created": "0.000000",
2014-01-23T03:56:55.180 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "epoch": 1,
2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "fsid": "ee15ef3b-5f6e-4cc1-bc0e-96436de15789",
2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "modified": "0.000000",
2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     "mons": [
2014-01-23T03:56:55.181 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       {
2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "addr": "10.214.138.87:6789/0",
2014-01-23T03:56:55.182 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "name": "vpm020",
2014-01-23T03:56:55.183 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "rank": 0
2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       },
2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       {
2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "addr": "10.214.138.112:6789/0",
2014-01-23T03:56:55.184 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "name": "vpm046",
2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "rank": 1
2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       },
2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       {
2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "addr": "0.0.0.0:0/2",
2014-01-23T03:56:55.185 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "name": "vpm077",
2014-01-23T03:56:55.187 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]         "rank": 2
2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]       }
2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]     ]
2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   },
2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "name": "vpm077",
2014-01-23T03:56:55.188 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "outside_quorum": [],
2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "quorum": [],
2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "rank": -1,
2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "state": "electing",
2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ]   "sync_provider": []
2014-01-23T03:56:55.189 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] }
2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;34mDEBUG[0m ] ********************************************************************************
2014-01-23T03:56:55.190 INFO:teuthology.orchestra.run.err:[10.214.138.87]: [[1mvpm077[0m][[1;37mINFO[0m  ] monitor: mon.vpm077 is not running

This causes health errors further along the test:

2014-01-23T03:58:47.030 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health'
2014-01-23T03:58:47.341 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046
2014-01-23T03:58:57.341 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health'
2014-01-23T03:58:57.626 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046
2014-01-23T03:59:07.626 DEBUG:teuthology.orchestra.run:Running [10.214.138.87]: 'cd /home/ubuntu/cephtest && sudo ceph health'
2014-01-23T03:59:07.926 DEBUG:teuthology.task.ceph-deploy:Ceph health: HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no osds; 1 mons down, quorum 0,1 vpm020,vpm046

Log file: http://qa-proxy.ceph.com/teuthology/teuthology-2014-01-22_06:21:18-ceph-deploy-master-testing-basic-vps/48108/teuthology.log

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #5804: mon: binds to 0.0.0.0:6800something portResolvedJoao Eduardo Luis07/29/2013

Actions
Actions
Copy link
 #1

Updated by Alfredo Deza over 10 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Joao Eduardo Luis about 10 years ago

  • Project changed from teuthology to Ceph
  • Category set to Monitor
  • Status changed from New to Fix Under Review

patch bb863b73c45ce5592844c2c72028ef1cfd9647f8 ; pull request: https://github.com/ceph/ceph/pull/1236

Actions
Copy link
 #3

Updated by Sage Weil about 10 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #4

Updated by Sage Weil about 10 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Assignee deleted (Joao Eduardo Luis)
Actions
Copy link
 #5

Updated by Ian Colle about 10 years ago

  • Priority changed from Urgent to High
  • Backport set to emperor, dumpling
Actions
Copy link
 #6

Updated by Joao Eduardo Luis about 10 years ago

  • Assignee set to Joao Eduardo Luis
Actions
Copy link
 #7

Updated by Sage Weil about 10 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF