Project

General

Profile

Bug #57650

mon-stretch: reweighting an osd to a big number, then back to original causes uneven weights

Added by Kamoltat (Junior) Sirivadhna over 1 year ago. Updated 9 months ago.

Status:
In Progress
Priority:
Normal
Category:
Stretch Clusters
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Reweight an osd from 0.0900 to 0.7000
and then reweight back to 0.0900. Causes uneven weights between
two zones reported by root. Although, each zone says its correct weight.

According to root: zone iris is 11796 and zone pze is 11795.

    {
            "id": -1,
            "name": "default",
            "type_id": 11,
            "type_name": "root",
            "weight": 23591,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -5,
                    "weight": 11796,
                    "pos": 0
                },
                {
                    "id": -7,
                    "weight": 11795,
                    "pos": 1
                }
            ]
        },

        {
            "id": -5,
            "name": "iris",
            "type_id": 9,
            "type_name": "zone",
            "weight": 11796,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -9,
                    "weight": 5898,
                    "pos": 0
                },
                {
                    "id": -10,
                    "weight": 5898,
                    "pos": 1
                }
            ]
        },
      {
            "id": -7,
            "name": "pze",
            "type_id": 9,
            "type_name": "zone",
            "weight": 11796,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -11,
                    "weight": 5898,
                    "pos": 0
                },
                {
                    "id": -12,
                    "weight": 5898,
                    "pos": 1
                }
            ]
        },

History

#1 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

ceph osd tree:

ID   CLASS  WEIGHT   TYPE NAME            STATUS  REWEIGHT  PRI-AFF
 -1         0.35997  root default                                  
 -5         0.17999      zone iris                                 
 -9         0.09000          host node-2                           
  0    hdd  0.09000              osd.0        up   1.00000  1.00000
-10         0.09000          host node-3                           
  1    hdd  0.09000              osd.1        up   1.00000  1.00000
 -7         0.17998      zone pze                                  
-11         0.09000          host node-4                           
  2    hdd  0.09000              osd.2        up   1.00000  1.00000
-12         0.09000          host node-5                           
  3    hdd  0.09000              osd.3        up   1.00000  1.00000

#2 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

  • Description updated (diff)

#3 Updated by Radoslaw Zarzynski over 1 year ago

  • Status changed from New to In Progress

#4 Updated by Kamoltat (Junior) Sirivadhna 9 months ago

This some how only reproducible with re weight an osd from 0.0900 to 0.7000 and back to 0.0900.

This PR https://github.com/ceph/ceph/pull/48209 re weight an osd from 0.0900 to 0.0700 and back to 0.0900 without hitting the bug.

#5 Updated by Kamoltat (Junior) Sirivadhna 9 months ago

  • Subject changed from mon-stretch: reweighting an osd back to original causes uneven weights to mon-stretch: reweighting an osd to a big number, then back to original causes uneven weights

#6 Updated by Kamoltat (Junior) Sirivadhna 9 months ago

Think this has to do with how we are subtracting/adding weights to each crush bucket, I think a better way is to always sum the weights from the bottom rather relying on just subtracting/adding the diff of the weights to each bucket.

Also available in: Atom PDF