Project

General

Profile

Bug #6698

new osds added to the cluster not starting up due to crush lookup failure

Added by Tamilarasi muthamizhan over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version: next branch [0.72-rc1]

test setup : mira025, mira038

had a cluster running on master branch, upgraded it to next and trying to add new osds to the cluster. The disks[osd.3, osd.4] are prepared and mounted but not starting up.

Associated revisions

Revision 1bb5aadb (diff)
Added by Sage Weil over 10 years ago

upstart: fix ceph-crush-location default

Behave if it is not specified on ceph.conf. (init-ceph also falls back
to the default.)

Fixes: #6698
Signed-off-by: Sage Weil <>

History

#1 Updated by Tamilarasi muthamizhan over 10 years ago

ubuntu@mira025:~/ceph-dep/ceph-deploy$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       902G  5.7G  850G   1% /
udev            7.8G   12K  7.8G   1% /dev
tmpfs           3.2G  336K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.9G     0  7.9G   0% /run/shm
cgroup          7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb1       931G   43M  931G   1% /var/lib/ceph/osd/ceph-0
/dev/sdc1       931G   33M  931G   1% /var/lib/ceph/osd/ceph-3
ubuntu@mira025:~/ceph-dep/ceph-deploy$ sudo ceph osd tree
# id    weight    type name    up/down    reweight
-1    2.73    root default
-2    0.91        host mira025
0    0.91            osd.0    down    1    
-3    1.82        host mira038
1    0.91            osd.1    down    1    
2    0.91            osd.2    down    1    

3    0    osd.3    down    0    
4    0    osd.4    down    0    

ubuntu@mira025:~/ceph-dep/ceph-deploy$ sudo ceph -v
ceph version 0.72-rc1-83-g592a99c (592a99ccd8da2d7843ebb7ce6439566732580b7a)

#2 Updated by Sage Weil over 10 years ago

  • Status changed from New to Fix Under Review

#3 Updated by Sage Weil over 10 years ago

  • Assignee deleted (Alfredo Deza)

#4 Updated by Sage Weil over 10 years ago

  • Status changed from Fix Under Review to Resolved

#5 Updated by Tamilarasi muthamizhan over 10 years ago

  • Status changed from Resolved to In Progress

this bug was caused by crush lookup failure.

I see similar issue with one of my local clusters: mira047 and mira048.

#6 Updated by Ian Colle over 10 years ago

  • Subject changed from ceph-deploy: new osds added to the cluster is not starting up to new osds added to the cluster not starting up due to crush lookup failure
  • Category deleted (ceph-deploy)
  • Status changed from In Progress to 12
  • Assignee set to Dan Mick

#7 Updated by Dan Mick over 10 years ago

  • Status changed from 12 to Need More Info

I guess I'm not sure what the expected behavior is here; do we think that the OSDs should start, but not do much, until they're added to the crush map?

#8 Updated by Dan Mick over 10 years ago

Sage suspects this may have been fixed by 1bb5aadbca80eb46d5393bb1ad68af3da025e157

#9 Updated by Tamilarasi muthamizhan over 10 years ago

  • Status changed from Need More Info to Resolved

tried reproducing this issue by upgrading from dumpling to emperor and then trying to add osds to the cluster but it worked fine.
closing the bug for now.

Also available in: Atom PDF