Ceph » RADOS

Actually, the problem with the weight not updated on the class subtree is easily reproducible on a vstart cluster (see the details below). But it turned out, on my vstart cluster the problem looks rather cosmetic: although the weight is not updated on the class subtree (adonis~ssd bucket in my case) the new weight is actually used to distribute pgs (according to `ceph osd df`).

This is not exactly what we observed for our customer (running nautilus). In their case they were redeploying osds (two hosts at once) with "osd crush initial weight = 0" in the config. Then they used "ceph osd crush set|reweight-subtree" commands to set a non-zero weight but were observing that osds were still not used. And only after they did some modifications to the crush map (redeployed another osds or just created/deleted a fake bucket in the crush map) the osds started to be used. As we noticed that "ceph osd crush set|reweight-subtree" commands did not change the weight on the class subtree we decided it was the cause that the osds were not used in their case, and we just recommended them to use "ceph osd crush reweight" command that properly updates weight on all subtrees. Unfortunately they do not need to redeploy in near future so we can not check at the moment. And as I failed to reproduce the case locally right now I am not exactly sure the problem was due to not updated weight on the class subtree. I am going to dig this further and will ask the customer if it is ok to share details about their cluster (osdmap) here.

Instructions for vstart cluster:

adonis:~/ceph/ceph/build% ../src/vstart.sh -n
...
adonis:~/ceph/ceph/build% ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         0.29576  root default                              
-3         0.29576      host adonis                           
 0    ssd  0.09859          osd.0        up   1.00000  1.00000
 1    ssd  0.09859          osd.1        up   1.00000  1.00000
 2    ssd  0.09859          osd.2        up   1.00000  1.00000
adonis:~/ceph/ceph/build% ceph osd crush set 0 0.666 host=adonis
set item id 0 name 'osd.0' weight 0.666 at location {host=adonis} to crush map
adonis:~/ceph/ceph/build% ceph osd tree                         
ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         0.86316  root default                              
-3         0.86316      host adonis                           
 0    ssd  0.66599          osd.0        up   1.00000  1.00000
 1    ssd  0.09859          osd.1        up   1.00000  1.00000
 2    ssd  0.09859          osd.2        up   1.00000  1.00000
adonis:~/ceph/ceph/build% ceph osd crush dump
{
    ...
    "buckets": [
    ...
        {
            "id": -3,
            "name": "adonis",
            "type_id": 1,
            "type_name": "host",
            "weight": 56568,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 0,
                    "weight": 43646,
                    "pos": 0
                },
                {
                    "id": 1,
                    "weight": 6461,
                    "pos": 1
                },
                {
                    "id": 2,
                    "weight": 6461,
                    "pos": 2
                }
            ]
        },
        {
            "id": -4,
            "name": "adonis~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 19383,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 0,
                    "weight": 6461,
                    "pos": 0
                },
                {
                    "id": 1,
                    "weight": 6461,
                    "pos": 1
                },
                {
                    "id": 2,
                    "weight": 6461,
                    "pos": 2
                }
            ]
        }
    ],

BTW Mykola I would suggest using 'ceph osd crush reweight osd.N' (which works fine already) instead of the 'ceph osd crush set ...' syntax (which is harder to use and suffers from this bug)

Sage Weil wrote:

BTW Mykola I would suggest using 'ceph osd crush reweight osd.N' (which works fine already) instead of the 'ceph osd crush set ...' syntax (which is harder to use and suffers from this bug)

Yes, that this what we recommended the customer, as I wrote in the comment #2. And that is why I was not sure it was a bug for that low level commands. Thank you for fixing this!