Project

General

Profile

Bug #22814

mgr: balancer: "some osds belong to multiple subtrees"

Added by Konstantin Shalygin almost 2 years ago. Updated over 1 year ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
balancer module
Target version:
% Done:

0%

Source:
Community (user)
Tags:
balancer mgr
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

ceph balancer mode crush-compat
ceph balancer on

2018-01-26 18:41:54.932473 7f1a2b5a7700  0 mgr[balancer] error: some osds belong to multiple subtrees: {0: 1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1
, 8: 1, 9: 1, 10: 1, 11: 1, 12: 1, 13: 1, 14: 1, 15: 1, 16: 1, 17: 1, 18: 1, 19: 1, 20: 1, 21: 1, 22: 1, 23: 1, 27: 1, 31: 1, 32: 1, 33: 1, 34: 1, 35:
 1, 36: 1, 37: 1, 38: 1, 39: 1, 40: 1, 41: 1, 42: 1, 43: 1, 44: 1, 45: 1, 46: 1, 47: 1, 48: 1, 49: 1, 50: 1, 51: 1, 52: 1, 53: 1}
ID  CLASS WEIGHT    TYPE NAME                        STATUS REWEIGHT PRI-AFF 
-12         6.00000 root solid                                               
-14         6.00000     datacenter some_place_solid                          
-15         2.00000         rack rack2-solid                                 
-13         1.00000             host ceph-osd0-solid                         
 24  nvme   1.00000                 osd.24               up  1.00000 1.00000 
-19         1.00000             host ceph-osd2-solid                         
 26  nvme   1.00000                 osd.26               up  1.00000 1.00000 
-16         2.00000         rack rack3-solid                                 
-20         1.00000             host ceph-osd3-solid                         
 30  nvme   1.00000                 osd.30               up  1.00000 1.00000 
-22         1.00000             host ceph-osd5-solid                         
 29  nvme   1.00000                 osd.29               up  1.00000 1.00000 
-17         2.00000         rack rack4-solid                                 
-18         1.00000             host ceph-osd1-solid                         
 25  nvme   1.00000                 osd.25               up  1.00000 1.00000 
-21         1.00000             host ceph-osd4-solid                         
 28  nvme   1.00000                 osd.28               up  1.00000 1.00000 
 -1       222.00000 root default                                             
 -7       222.00000     datacenter some_place                                
 -8        74.00000         rack rack2-hdd                                   
 -2        37.00000             host ceph-osd0-hdd                           
  0   hdd   5.00000                 osd.0                up  1.00000 1.00000 
  4   hdd   4.00000                 osd.4                up  1.00000 1.00000 
  7   hdd   4.00000                 osd.7                up  1.00000 1.00000 
  9   hdd   4.00000                 osd.9                up  1.00000 1.00000 
 46   hdd   5.00000                 osd.46               up  1.00000 1.00000 
 47   hdd   5.00000                 osd.47               up  1.00000 1.00000 
 48   hdd   5.00000                 osd.48               up  1.00000 1.00000 
 49   hdd   5.00000                 osd.49               up  1.00000 1.00000 
 -6        37.00000             host ceph-osd2-hdd                           
 12   hdd   5.00000                 osd.12               up  1.00000 1.00000 
 14   hdd   4.00000                 osd.14               up  1.00000 1.00000 
 18   hdd   4.00000                 osd.18               up  1.00000 1.00000 
 19   hdd   4.00000                 osd.19               up  1.00000 1.00000 
 50   hdd   5.00000                 osd.50               up  1.00000 1.00000 
 51   hdd   5.00000                 osd.51               up  1.00000 1.00000 
 52   hdd   5.00000                 osd.52               up  1.00000 1.00000 
 53   hdd   5.00000                 osd.53               up  1.00000 1.00000 
 -9        74.00000         rack rack3-hdd                                   
 -3        37.00000             host ceph-osd3-hdd                           
 20   hdd   5.00000                 osd.20               up  1.00000 1.00000 
 21   hdd   4.00000                 osd.21               up  1.00000 1.00000 
 22   hdd   4.00000                 osd.22               up  1.00000 1.00000 
 23   hdd   4.00000                 osd.23               up  1.00000 1.00000 
 34   hdd   5.00000                 osd.34               up  1.00000 1.00000 
 35   hdd   5.00000                 osd.35               up  1.00000 1.00000 
 36   hdd   5.00000                 osd.36               up  1.00000 1.00000 
 37   hdd   5.00000                 osd.37               up  1.00000 1.00000 
-11        37.00000             host ceph-osd5-hdd                           
 13   hdd   5.00000                 osd.13               up  1.00000 1.00000 
 15   hdd   4.00000                 osd.15               up  1.00000 1.00000 
 16   hdd   4.00000                 osd.16               up  1.00000 1.00000 
 17   hdd   4.00000                 osd.17               up  1.00000 1.00000 
 27   hdd   5.00000                 osd.27               up  1.00000 1.00000 
 31   hdd   5.00000                 osd.31               up  1.00000 1.00000 
 32   hdd   5.00000                 osd.32               up  1.00000 1.00000 
 33   hdd   5.00000                 osd.33               up  1.00000 1.00000 
-10        74.00000         rack rack4-hdd                                   
 -4        37.00000             host ceph-osd1-hdd                           
  1   hdd   5.00000                 osd.1                up  1.00000 1.00000 
  2   hdd   4.00000                 osd.2                up  1.00000 1.00000 
  3   hdd   4.00000                 osd.3                up  1.00000 1.00000 
  5   hdd   4.00000                 osd.5                up  1.00000 1.00000 
 38   hdd   5.00000                 osd.38               up  1.00000 1.00000 
 39   hdd   5.00000                 osd.39               up  1.00000 1.00000 
 40   hdd   5.00000                 osd.40               up  1.00000 1.00000 
 41   hdd   5.00000                 osd.41               up  1.00000 1.00000 
 -5        37.00000             host ceph-osd4-hdd                           
  6   hdd   5.00000                 osd.6                up  1.00000 1.00000 
  8   hdd   4.00000                 osd.8                up  1.00000 1.00000 
 10   hdd   4.00000                 osd.10               up  1.00000 1.00000 
 11   hdd   4.00000                 osd.11               up  1.00000 1.00000 
 42   hdd   5.00000                 osd.42               up  1.00000 1.00000 
 43   hdd   5.00000                 osd.43               up  1.00000 1.00000 
 44   hdd   5.00000                 osd.44               up  1.00000 1.00000 
 45   hdd   5.00000                 osd.45               up  1.00000 1.00000 

Multi root is a problem for balancer?

balancer module from 'master', cluster version 12.2.2

History

#1 Updated by Sage Weil over 1 year ago

  • Status changed from New to Need More Info

Hi Konstantin,

This error is confusing because you don't see to have osds under multiple roots from the dump. Do you still see this symptom? If you could attach your osdmap that would be helpful!

(You do have parallel trees, one for -solid, which is no longer necessary now that device classes are in place, though; you might consider switching over to using that. Note that it currently will shuffle some data around; if that's a problem we're working on a way to mitigate that during the transition.)

#2 Updated by Konstantin Shalygin over 1 year ago

Hi Sage,

I migrated to single root already. So I don't have this osdmap. But I have mon leveldb backups from this day - can this help?

Also available in: Atom PDF