Ceph

In trying to develop CRUSH rulesets, I've noticed that crushtool --test --simulate does not follow the rules in the CRUSH map properly. I'm attaching a new map that has some slightly different issues than what was provided in the email. Here is a transcript of the e-mail I sent to the mailing list:

==== Email ====
I'm trying to create a CRUSH ruleset and I'm using crushtool to test
the rules, but it doesn't seem to mapping things correctly. I have two
roots, one for spindles and another for SSD. I have two rules, one for
each root. The output of crushtool on rule 0 shows objects being
mapped to SSD OSDs when it should only be choosing spindles.

I'm pretty sure I'm doing something wrong. I've tested the map on .93 and .80.8.

The map is at http://pastebin.com/BjmuASX0

when running

crushtool -i map.crush --test --num-rep 3 --rule 0 --simulate --show-mappings

I'm getting mapping to OSDs > 39 which are SSDs. The same happens when
I run the SSD rule, I get OSDs from both roots. It is as if crushtool
is not selecting the correct root. In fact both rules result in the
same mapping:

RNG rule 0 x 0 [0,38,23]
RNG rule 0 x 1 [10,25,1]
RNG rule 0 x 2 [11,40,0]
RNG rule 0 x 3 [5,30,26]
RNG rule 0 x 4 [44,30,10]
RNG rule 0 x 5 [8,26,16]
RNG rule 0 x 6 [24,5,36]
RNG rule 0 x 7 [38,10,9]
RNG rule 0 x 8 [39,9,23]
RNG rule 0 x 9 [12,3,24]
RNG rule 0 x 10 [18,6,41]
...

RNG rule 1 x 0 [0,38,23]
RNG rule 1 x 1 [10,25,1]
RNG rule 1 x 2 [11,40,0]
RNG rule 1 x 3 [5,30,26]
RNG rule 1 x 4 [44,30,10]
RNG rule 1 x 5 [8,26,16]
RNG rule 1 x 6 [24,5,36]
RNG rule 1 x 7 [38,10,9]
RNG rule 1 x 8 [39,9,23]
RNG rule 1 x 9 [12,3,24]
RNG rule 1 x 10 [18,6,41]
...

==== Email ====

In addition, I modified the rules to do multiple selects (select 2 racks, then 2 hosts in each rack) and it is not working properly. It only returns two OSDs. The CRUSH rules work fine when imported into the cluster. For some reason, rule 0 this version of the CRUSH is not selecting OSDs > 39 like it was previously, but rule 1 is selecting OSDs < 40 which it shouldn't and none > 39 which it should.

[root@nodezz ~]# crushtool -c crush.test -o map.test; crushtool -i map.test --test --num-rep 3 --simulate --show-mappings --max-x 10 --min-x 0 --show-statistics
rule 0 (replicated_ruleset), x = 0..10, numrep = 3..3
RNG rule 0 x 0 [0,38,15]
RNG rule 0 x 1 [26,6,36]
RNG rule 0 x 2 [9,21,12]
RNG rule 0 x 3 [26,33,1]
RNG rule 0 x 4 [39,6,15]
RNG rule 0 x 5 [35,2,10]
RNG rule 0 x 6 [35,0,24]
RNG rule 0 x 7 [37,6,26]
RNG rule 0 x 8 [4,38,18]
RNG rule 0 x 9 [37,26,4]
RNG rule 0 x 10 [39,2,15]
rule 0 (replicated_ruleset) num_rep 3 result size == 3: 11/11
rule 1 (test_ruleset), x = 0..10, numrep = 3..3
RNG rule 1 x 0 [16,34]
RNG rule 1 x 1 [8,30]
RNG rule 1 x 2 [16,39]
RNG rule 1 x 3 [25,7]
RNG rule 1 x 4 [0,30]
RNG rule 1 x 5 [23,12]
RNG rule 1 x 6 [27,0]
RNG rule 1 x 7 [34,6]
RNG rule 1 x 8 [17,31]
RNG rule 1 x 9 [17,36]
RNG rule 1 x 10 [9,23]
rule 1 (test_ruleset) num_rep 3 result size == 2: 11/11