Ceph

h1. Crush extension for more flexible object placement

h3. Summary

Extend crush to allow more flexible object placement

h3. Owners

* Li Wang (liwang@ubuntukylin.com)

* Lianghao Shen (lianghaoshen@ubuntukylin.com)

h3. Interested Parties

* Name (Affiliation)

* Name (Affiliation)

* Name

h3. Current Status

h3. Detailed Description

This blueprint is originally proposed by Sage at https://wiki.ceph.com/Planning/Blueprints/Dumpling/extend_crush_rule_language.

CRUSH is a deterministic, rule-based consistent hash-like algorithm (with some very nice properties) for determining object placement in distributed storage systems. Its selection rule language, while already very useful, is incapable of expressing some useful rules:

 - The current choose iterates over the 'working set' and recursively selects new items for each item.  It always applies to all items in the working set.  That precludes strategies like "pick 2 racks, choose N from the first, and M from the second".

 - It is assumed the hierarchy is a single uniform tree.  You cannot have two parallel trees of devices (say, SSDs and HDDs) in the same nodes, and pick 1 ssd and 1 hdd but ensure that they exist in different hosts.

Algorithm:

Some new features should be developed for these above scenarios:

 - Unsymmetric choose option, e.g., _assign(N,TYPE1,TYPE2,a1,a2...an)_, which supports choosing _N TYPE1_ buckets, and then choosing _ai(i=1,2..N) TYPE2_ buckets from the _N TYPE1_ buckets, respectively.

<pre>

assign(n,type1,type2,a1,a2...an) {

    map = [a1,a2, ..., an];

    out1 = choose_firstn (n, type1, ...);  // choose n items with a type of type1

    for items m in out1 {

        sub_item = choose_firstn(map[m], type2, ...);

        out2 = out2+sub_item;

      }

}

</pre>

Osds grouping, _group {devicetype|network}_, which supports identifying the osds with groupids according to the given strategy, namely, by devicetype or network.

 - Modified chooseleaf option, _chooseleaf firstn {num} type {bucket-type} [group]_, which can support placing the replicas in different groups, relative to the original one.

<pre>

group {devicetype|network...}

    for osds in pool：

    if osd.device == ssd

        osd.gid = 1;

    else

        osd.gid = 2;

choose firstn n type osd [group]   

    item = crush_bucket_choose(in, x, r);

    if(group){

        if (is_gid_selected(out,item)){

            fgroup++;       

            retry_group = 1;

        }

    }

</pre>

h3. Work items

h4. Coding tasks

# Task 1

# Task 2

# Task 3

h4. Build / release tasks

# Task 1

# Task 2

# Task 3

h4. Documentation tasks

# Task 1

# Task 2

# Task 3

h4. Deprecation tasks

# Task 1

# Task 2

# Task 3