Project

General

Profile

Bug #6698

new osds added to the cluster not starting up due to crush lookup failure

Added by Tamilarasi muthamizhan almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
Start date:
10/31/2013
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

ceph version: next branch [0.72-rc1]

test setup : mira025, mira038

had a cluster running on master branch, upgraded it to next and trying to add new osds to the cluster. The disks[osd.3, osd.4] are prepared and mounted but not starting up.

Associated revisions

Revision 1bb5aadb (diff)
Added by Sage Weil almost 4 years ago

upstart: fix ceph-crush-location default

Behave if it is not specified on ceph.conf. (init-ceph also falls back
to the default.)

Fixes: #6698
Signed-off-by: Sage Weil <>

History

#1 Updated by Tamilarasi muthamizhan almost 4 years ago

ubuntu@mira025:~/ceph-dep/ceph-deploy$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       902G  5.7G  850G   1% /
udev            7.8G   12K  7.8G   1% /dev
tmpfs           3.2G  336K  3.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.9G     0  7.9G   0% /run/shm
cgroup          7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb1       931G   43M  931G   1% /var/lib/ceph/osd/ceph-0
/dev/sdc1       931G   33M  931G   1% /var/lib/ceph/osd/ceph-3
ubuntu@mira025:~/ceph-dep/ceph-deploy$ sudo ceph osd tree
# id    weight    type name    up/down    reweight
-1    2.73    root default
-2    0.91        host mira025
0    0.91            osd.0    down    1    
-3    1.82        host mira038
1    0.91            osd.1    down    1    
2    0.91            osd.2    down    1    

3    0    osd.3    down    0    
4    0    osd.4    down    0    

ubuntu@mira025:~/ceph-dep/ceph-deploy$ sudo ceph -v
ceph version 0.72-rc1-83-g592a99c (592a99ccd8da2d7843ebb7ce6439566732580b7a)

#2 Updated by Sage Weil almost 4 years ago

  • Status changed from New to Need Review

#3 Updated by Sage Weil almost 4 years ago

  • Assignee deleted (Alfredo Deza)

#4 Updated by Sage Weil almost 4 years ago

  • Status changed from Need Review to Resolved

#5 Updated by Tamilarasi muthamizhan almost 4 years ago

  • Status changed from Resolved to In Progress

this bug was caused by crush lookup failure.

I see similar issue with one of my local clusters: mira047 and mira048.

#6 Updated by Ian Colle almost 4 years ago

  • Subject changed from ceph-deploy: new osds added to the cluster is not starting up to new osds added to the cluster not starting up due to crush lookup failure
  • Category deleted (ceph-deploy)
  • Status changed from In Progress to Verified
  • Assignee set to Dan Mick

#7 Updated by Dan Mick almost 4 years ago

  • Status changed from Verified to Need More Info

I guess I'm not sure what the expected behavior is here; do we think that the OSDs should start, but not do much, until they're added to the crush map?

#8 Updated by Dan Mick almost 4 years ago

Sage suspects this may have been fixed by 1bb5aadbca80eb46d5393bb1ad68af3da025e157

#9 Updated by Tamilarasi muthamizhan almost 4 years ago

  • Status changed from Need More Info to Resolved

tried reproducing this issue by upgrading from dumpling to emperor and then trying to add osds to the cluster but it worked fine.
closing the bug for now.

Also available in: Atom PDF