Ceph

Hi

I have worked for some time with ceph, and had weird problem for few days. First my configuration:
host01: osd1 (local disk), osd2 (local disk), osd3 (local disk) ... osd11 (local disk)
host02: osd20 (local disk), osd21 (local disk), osd22 (local disk) ... osd31 (local disk)

so 24 OSD's (each of one disk), on two hosts

and my size = 3, and my min_size = 2
pg number was 1200 (yes, I know about power 2, but because of some tests, and no possibility to decrease this number it must be as it is for now)

and whatever I have done, none even one time I had active+clean state, and after time spent looking into documentation, and people examples etc. my collegue found this:

The number of replica's you want must be covered by at least the same number of items of several 'types' in the crushmap hierarchie. For instance if you want replica count of 3 you must make sure to have atleast 3 'hosts' in the ceph crush map. Since those are only names of hierarchie layers, those 3 hosts could actually map to the same server if you really wanted to. See here for reference: http://forum.proxmox.com/threads/20695-CEPH-crush-map-and-CEPH-cluster-HEALTH

ITEMS of several types !

so I changed my CRUSH from this:
host md002 {
id -2 # do not change unnecessarily # weight 21.840
alg straw
hash 0 # rjenkins1
item osd.20 weight 1.820
item osd.21 weight 1.820
item osd.22 weight 1.820
item osd.23 weight 1.820
item osd.24 weight 1.820
item osd.25 weight 1.820
item osd.26 weight 1.820
item osd.27 weight 1.820
item osd.28 weight 1.820
item osd.29 weight 1.820
item osd.30 weight 1.820
item osd.31 weight 1.820
}
host md008 {
id -3 # do not change unnecessarily # weight 21.840
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.820
item osd.1 weight 1.820
item osd.2 weight 1.820
item osd.3 weight 1.820
item osd.4 weight 1.820
item osd.5 weight 1.820
item osd.6 weight 1.820
item osd.7 weight 1.820
item osd.8 weight 1.820
item osd.9 weight 1.820
item osd.10 weight 1.820
item osd.11 weight 1.820
}
root default {
id -1 # do not change unnecessarily # weight 43.680
alg straw
hash 0 # rjenkins1
item md002 weight 21.840
item md008 weight 21.840
}

to this:

host md002 {
id -2 # do not change unnecessarily
alg straw
hash 0 # rjenkins1
item osd.20 weight 1.000
item osd.21 weight 1.000
item osd.22 weight 1.000
item osd.23 weight 1.000
item osd.24 weight 1.000
item osd.25 weight 1.000
item osd.26 weight 1.000
item osd.27 weight 1.000
}
host md008 {
id -3 # do not change unnecessarily # weight 12.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.000
item osd.1 weight 1.000
item osd.2 weight 1.000
item osd.3 weight 1.000
item osd.4 weight 1.000
item osd.5 weight 1.000
item osd.6 weight 1.000
item osd.7 weight 1.000
}

host combined {
id -4
alg straw
item osd.8 weight 1.000
item osd.9 weight 1.000
item osd.10 weight 1.000
item osd.11 weight 1.000
item osd.28 weight 1.000
item osd.29 weight 1.000
item osd.30 weight 1.000
item osd.31 weight 1.000
}

root default {
id -1 # do not change unnecessarily # weight 43.680
alg straw
hash 0 # rjenkins1
item md002 weight 1.000
item md008 weight 1.000
item combined weight 1.000
}

and ... i t looks it works

and this is my suggestion to add an information to documentation

direct relation between osd numbers, pg numbers, replicas, and types of crushmap hierarchy

maybe even there is a better configuration of our cluster that what we "combined"