Bug #22431
closedbalancer: calc_pg_upmaps not upmapping the correct pool
0%
Description
On this cluster, only the first pool "test" was ever upmapped:
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
test 2 0 0 105T 0
cephfs_data 3 0 0 105T 0
cephfs_metadata 4 3558 0 105T 39Here is a log from the balancer, where it should have upmapped pool 3, but chose 2 instead:
2017-12-13 16:50:35.201425 7fd6f2063700 4 mgr[balancer] pools ['cephfs_data', 'cephfs_metadata', 'test']
2017-12-13 16:50:36.450845 7fd6f2063700 4 mgr[balancer] prepared 10/10 changes
2017-12-13 16:50:36.451232 7fd6f2063700 4 mgr[balancer] Executing plan auto_2017-12-13_15:50:33
2017-12-13 16:50:36.453830 7fd6f2063700 4 mgr[balancer] ceph osd rm-pg-upmap-items 2.b7
2017-12-13 16:50:36.454020 7fd6f2063700 4 mgr[balancer] ceph osd rm-pg-upmap-items 2.2aa
2017-12-13 16:50:36.454150 7fd6f2063700 4 mgr[balancer] ceph osd rm-pg-upmap-items 2.2ef
2017-12-13 16:50:36.454263 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.73 mappings [{'to': 307L, 'from': 292L}]
2017-12-13 16:50:36.454396 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.b7 mappings [{'to': 0L, 'from': 426L}]
2017-12-13 16:50:36.454584 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.2aa mappings [{'to': 294L, 'from': 519L},
{'to': 39L, 'from': 89L}, {'to': 180L, 'from': 240L}]
2017-12-13 16:50:36.454697 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.334 mappings [{'to': 47L, 'from': 198L}, {
'to': 13L, 'from': 298L}]
2017-12-13 16:50:36.454796 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.719 mappings [{'to': 0L, 'from': 287L}]
2017-12-13 16:50:36.454894 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.90b mappings [{'to': 0L, 'from': 221L}]
2017-12-13 16:50:36.454991 7fd6f2063700 4 mgr[balancer] ceph osd pg-upmap-items 2.970 mappings [{'to': 2L, 'from': 280L}]
Updated by Sage Weil over 6 years ago
- Status changed from New to Need More Info
So, a couple of possible causes:
- upmap treats all pgs the same. it doesn't factor in how well populated the pool(s) are--it just balances pg counts. We can fix that by doing something similar to what the crush-compat balancer code does where it combines pg counts, object counts, and/or bytes?
- it's possibly only a small number of upmaps were done for that first iteration.. once the cluster stabilizes it should work on another random pool, right? at the end of the day it shouldn't converge until all pools' pgs are balanced
Updated by Dan van der Ster over 6 years ago
- upmap treats all pgs the same. it doesn't factor in how well populated the pool(s) are--it just balances pg counts. We can fix that by doing something similar to what the crush-compat balancer code does where it combines pg counts, object counts, and/or bytes?
Oh yes, I hadn't realized that. Indeed upmap will need similar smarts.
- it's possibly only a small number of upmaps were done for that first iteration.. once the cluster stabilizes it should work on another random pool, right? at the end of the day it shouldn't converge until all pools' pgs are balanced
This is where I'm confused. In my example, the random.shuffle(pools) has put pool 3 first. So the first call to osdmap.calc_pg_upmaps should pass pool=3, which should plan some moves. But for some reason it's not moving pool 3, it only ever moves pool 2. This cluster already has 1228 pg_upmap_items, and all are for pool 2.
Could it be that the [pool] arg is not passed correctly, so osdmap.calc_pg_upmaps is proceeding as if only_pools_orig.empty() is true?
Updated by Dan van der Ster almost 6 years ago
This can probably be closed -- we're not able to reproduce it anymore.
Updated by Konstantin Shalygin almost 3 years ago
- Status changed from Need More Info to Can't reproduce