Bug #41313
PG distribution completely messed up since Nautilus
0%
Description
Since using Nautilus the data distribution is an absolute nightmare. We never had issues on Luminous.
Look at this please. I added those empty disks yesterday and nothing happened (rebalance is running but not sure its using them). The worst is OSD 15, it fills up the most, I already had to set it weight to 0 and its still filling it up. I had to down/out it. This should never happen.
And no that PG autobalancer is off because it doens't work whilst cluster is in ERR/WARN state anyway. Not sure what changed in the code but its BAD.
14 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 125 KiB 13 GiB 3.3 TiB 63.95 2.10 112 up 19 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 88 KiB 12 GiB 3.5 TiB 61.50 2.02 102 up 22 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 335 KiB 14 GiB 2.9 TiB 68.32 2.24 121 up 25 hdd 9.09470 1.00000 9.1 TiB 5.1 TiB 5.1 TiB 96 KiB 11 GiB 4.0 TiB 56.07 1.84 97 up 30 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 64 KiB 11 GiB 4.2 TiB 53.89 1.77 89 up 33 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 57 KiB 12 GiB 3.5 TiB 62.06 2.03 105 up 34 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 44 KiB 14 GiB 2.9 TiB 67.93 2.23 121 up 35 hdd 9.09470 1.00000 9.1 TiB 3.9 TiB 3.9 TiB 19 KiB 9.2 GiB 5.2 TiB 43.24 1.42 71 up 12 hdd 9.09470 1.00000 9.1 TiB 5.3 TiB 5.3 TiB 56 KiB 12 GiB 3.8 TiB 58.45 1.92 100 up 16 hdd 9.09470 1.00000 9.1 TiB 4.2 TiB 4.2 TiB 28 KiB 9.8 GiB 4.9 TiB 46.62 1.53 79 up 17 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 108 KiB 14 GiB 2.7 TiB 70.81 2.32 119 up 20 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 48 KiB 13 GiB 3.3 TiB 63.82 2.09 105 up 23 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 60 KiB 13 GiB 3.4 TiB 62.16 2.04 107 up 26 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 195 KiB 14 GiB 2.4 TiB 73.69 2.42 131 up 28 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 52 KiB 12 GiB 3.4 TiB 62.19 2.04 109 up 31 hdd 9.09470 1.00000 9.1 TiB 3.8 TiB 3.8 TiB 41 KiB 9.0 GiB 5.3 TiB 41.68 1.37 69 up 13 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.4 TiB 32 KiB 12 GiB 3.6 TiB 59.91 1.96 102 up 15 hdd 0 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 112 KiB 16 GiB 1.5 TiB 83.69 2.74 147 up 18 hdd 9.09470 1.00000 9.1 TiB 4.7 TiB 4.7 TiB 198 KiB 10 GiB 4.4 TiB 51.60 1.69 87 up 21 hdd 9.09470 1.00000 9.1 TiB 4.3 TiB 4.3 TiB 4 KiB 10 GiB 4.8 TiB 47.49 1.56 77 up 24 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 64 KiB 14 GiB 2.8 TiB 68.91 2.26 122 up 27 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 298 KiB 13 GiB 3.1 TiB 65.78 2.16 113 up 29 hdd 9.09470 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 147 KiB 11 GiB 3.9 TiB 57.38 1.88 102 up 32 hdd 9.09470 1.00000 9.1 TiB 4.6 TiB 4.6 TiB 317 KiB 10 GiB 4.5 TiB 50.64 1.66 84 up 37 hdd 9.09470 1.00000 9.1 TiB 2.1 TiB 2.1 TiB 32 KiB 5.0 GiB 7.0 TiB 22.62 0.74 31 up 39 hdd 9.09470 1.00000 9.1 TiB 1.8 TiB 1.8 TiB 88 KiB 4.2 GiB 7.3 TiB 19.32 0.63 29 up 41 hdd 9.09470 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 44 KiB 4.5 GiB 7.2 TiB 21.13 0.69 32 up 43 hdd 9.09470 1.00000 9.1 TiB 2.8 TiB 2.8 TiB 56 KiB 6.3 GiB 6.3 TiB 30.67 1.01 43 up 45 hdd 9.09470 1.00000 9.1 TiB 2.9 TiB 2.9 TiB 48 KiB 6.8 GiB 6.2 TiB 32.09 1.05 53 up 46 hdd 9.09470 1.00000 9.1 TiB 2.7 TiB 2.7 TiB 48 KiB 6.3 GiB 6.4 TiB 29.54 0.97 45 up 48 hdd 9.09470 1.00000 9.1 TiB 2.5 TiB 2.5 TiB 44 KiB 5.7 GiB 6.6 TiB 27.77 0.91 41 up 50 hdd 9.09470 1.00000 9.1 TiB 1.9 TiB 1.9 TiB 76 KiB 4.4 GiB 7.2 TiB 21.01 0.69 28 up 36 hdd 9.09470 1.00000 9.1 TiB 1.7 TiB 1.7 TiB 32 KiB 4.1 GiB 7.4 TiB 18.77 0.62 31 up 38 hdd 9.09470 1.00000 9.1 TiB 2.3 TiB 2.3 TiB 28 KiB 5.5 GiB 6.8 TiB 25.44 0.83 37 up 40 hdd 9.09470 1.00000 9.1 TiB 2.2 TiB 2.2 TiB 56 KiB 5.1 GiB 6.9 TiB 24.52 0.80 40 up 42 hdd 9.09470 1.00000 9.1 TiB 2.3 TiB 2.2 TiB 60 KiB 5.2 GiB 6.8 TiB 24.78 0.81 34 up 44 hdd 9.09470 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 36 KiB 5.9 GiB 6.5 TiB 28.75 0.94 46 up 47 hdd 9.09470 1.00000 9.1 TiB 2.6 TiB 2.6 TiB 64 KiB 5.9 GiB 6.5 TiB 28.69 0.94 42 up 49 hdd 9.09470 1.00000 9.1 TiB 2.1 TiB 2.1 TiB 52 KiB 4.8 GiB 7.0 TiB 22.79 0.75 34 up 51 hdd 9.09470 1.00000 9.1 TiB 2.1 TiB 2.1 TiB 36 KiB 4.9 GiB 7.0 TiB 23.44 0.77 35 up 52 hdd 9.09470 1.00000 9.1 TiB 15 GiB 14 GiB 28 KiB 1024 MiB 9.1 TiB 0.17 0.01 0 up 53 hdd 9.09470 1.00000 9.1 TiB 20 GiB 19 GiB 4 KiB 1024 MiB 9.1 TiB 0.22 0.01 0 up 54 hdd 9.09470 1.00000 9.1 TiB 11 GiB 10 GiB 28 KiB 1024 MiB 9.1 TiB 0.12 0.00 0 up 55 hdd 9.09470 1.00000 9.1 TiB 17 GiB 16 GiB 4 KiB 1024 MiB 9.1 TiB 0.18 0.01 0 up 56 hdd 9.09470 1.00000 9.1 TiB 19 GiB 18 GiB 8 KiB 1024 MiB 9.1 TiB 0.20 0.01 0 up 57 hdd 9.09470 1.00000 9.1 TiB 21 GiB 20 GiB 4 KiB 1024 MiB 9.1 TiB 0.22 0.01 0 up 58 hdd 9.09470 1.00000 9.1 TiB 21 GiB 20 GiB 12 KiB 1024 MiB 9.1 TiB 0.22 0.01 0 up 59 hdd 9.09470 1.00000 9.1 TiB 21 GiB 20 GiB 20 KiB 1024 MiB 9.1 TiB 0.22 0.01 0 up 60 hdd 9.09470 1.00000 9.1 TiB 19 GiB 18 GiB 4 KiB 1024 MiB 9.1 TiB 0.21 0.01 0 up 61 hdd 9.09470 1.00000 9.1 TiB 13 GiB 12 GiB 20 KiB 1024 MiB 9.1 TiB 0.14 0.00 0 up 62 hdd 9.09470 1.00000 9.1 TiB 19 GiB 18 GiB 28 KiB 1024 MiB 9.1 TiB 0.20 0.01 0 up 63 hdd 9.09470 1.00000 9.1 TiB 20 GiB 19 GiB 16 KiB 1024 MiB 9.1 TiB 0.21 0.01 0 up 64 hdd 9.09470 1.00000 9.1 TiB 19 GiB 18 GiB 28 KiB 1024 MiB 9.1 TiB 0.20 0.01 0 up 65 hdd 9.09470 1.00000 9.1 TiB 19 GiB 18 GiB 20 KiB 1024 MiB 9.1 TiB 0.21 0.01 0 up 66 hdd 9.09470 1.00000 9.1 TiB 20 GiB 19 GiB 8 KiB 1024 MiB 9.1 TiB 0.21 0.01 0 up 67 hdd 9.09470 1.00000 9.1 TiB 20 GiB 19 GiB 12 KiB 1024 MiB 9.1 TiB 0.22 0.01 0 up
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 542.11450 root default -19 72.75757 host fsn-1-dc4-s3-1035753 14 hdd 9.09470 osd.14 up 1.00000 1.00000 19 hdd 9.09470 osd.19 up 1.00000 1.00000 22 hdd 9.09470 osd.22 up 1.00000 1.00000 25 hdd 9.09470 osd.25 up 1.00000 1.00000 30 hdd 9.09470 osd.30 up 1.00000 1.00000 33 hdd 9.09470 osd.33 up 1.00000 1.00000 34 hdd 9.09470 osd.34 up 1.00000 1.00000 35 hdd 9.09470 osd.35 up 1.00000 1.00000 -16 72.75757 host fsn-1-dc4-s3-1060345 12 hdd 9.09470 osd.12 up 1.00000 1.00000 16 hdd 9.09470 osd.16 up 1.00000 1.00000 17 hdd 9.09470 osd.17 up 1.00000 1.00000 20 hdd 9.09470 osd.20 up 1.00000 1.00000 23 hdd 9.09470 osd.23 up 1.00000 1.00000 26 hdd 9.09470 osd.26 up 1.00000 1.00000 28 hdd 9.09470 osd.28 up 1.00000 1.00000 31 hdd 9.09470 osd.31 up 1.00000 1.00000 -13 63.66287 host fsn-1-dc4-s3-1060346 13 hdd 9.09470 osd.13 up 1.00000 1.00000 15 hdd 0 osd.15 up 1.00000 1.00000 18 hdd 9.09470 osd.18 up 1.00000 1.00000 21 hdd 9.09470 osd.21 up 1.00000 1.00000 24 hdd 9.09470 osd.24 up 1.00000 1.00000 27 hdd 9.09470 osd.27 up 1.00000 1.00000 29 hdd 9.09470 osd.29 up 1.00000 1.00000 32 hdd 9.09470 osd.32 up 1.00000 1.00000 -22 72.75757 host fsn-1-dc4-s3-1065722 37 hdd 9.09470 osd.37 up 1.00000 1.00000 39 hdd 9.09470 osd.39 up 1.00000 1.00000 41 hdd 9.09470 osd.41 up 1.00000 1.00000 43 hdd 9.09470 osd.43 up 1.00000 1.00000 45 hdd 9.09470 osd.45 up 1.00000 1.00000 46 hdd 9.09470 osd.46 up 1.00000 1.00000 48 hdd 9.09470 osd.48 up 1.00000 1.00000 50 hdd 9.09470 osd.50 up 1.00000 1.00000 -25 72.75757 host fsn-1-dc4-s3-1065723 36 hdd 9.09470 osd.36 up 1.00000 1.00000 38 hdd 9.09470 osd.38 up 1.00000 1.00000 40 hdd 9.09470 osd.40 up 1.00000 1.00000 42 hdd 9.09470 osd.42 up 1.00000 1.00000 44 hdd 9.09470 osd.44 up 1.00000 1.00000 47 hdd 9.09470 osd.47 up 1.00000 1.00000 49 hdd 9.09470 osd.49 up 1.00000 1.00000 51 hdd 9.09470 osd.51 up 1.00000 1.00000 -28 72.75757 host fsn-1-dc4-s3-1066329 52 hdd 9.09470 osd.52 up 1.00000 1.00000 53 hdd 9.09470 osd.53 up 1.00000 1.00000 54 hdd 9.09470 osd.54 up 1.00000 1.00000 55 hdd 9.09470 osd.55 up 1.00000 1.00000 56 hdd 9.09470 osd.56 up 1.00000 1.00000 57 hdd 9.09470 osd.57 up 1.00000 1.00000 58 hdd 9.09470 osd.58 up 1.00000 1.00000 59 hdd 9.09470 osd.59 up 1.00000 1.00000 -31 72.75757 host fsn-1-dc4-s3-1066330 60 hdd 9.09470 osd.60 up 1.00000 1.00000 61 hdd 9.09470 osd.61 up 1.00000 1.00000 62 hdd 9.09470 osd.62 up 1.00000 1.00000 63 hdd 9.09470 osd.63 up 1.00000 1.00000 64 hdd 9.09470 osd.64 up 1.00000 1.00000 65 hdd 9.09470 osd.65 up 1.00000 1.00000 66 hdd 9.09470 osd.66 up 1.00000 1.00000 67 hdd 9.09470 osd.67 up 1.00000 1.00000
pgs: 6515347/169178931 objects degraded (3.851%) 77130467/169178931 objects misplaced (45.591%) 652 active+remapped+backfill_wait 322 active+clean 89 active+undersized+degraded+remapped+backfill_wait 58 active+undersized+degraded+remapped+backfilling 54 active+remapped+backfill_wait+backfill_toofull 33 active+remapped+backfilling
History
#1 Updated by Neha Ojha over 4 years ago
Can you reach out on the ceph-users mailing list to see if others have seen similar issues? We've not seen a specific bug related to this.
#2 Updated by Anonymous over 4 years ago
There definitely is something wrong.
After every kind of rebalance, no matter if caused by OSD removal/adding, or now PG raise, the PG distribution gets worse and worse. I was raising pool PGs every day bit by bit. First it was 1 nearfull OSD, then 4, now it's 8!.
14 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 16 GiB 3.5 TiB 61.34 0.86 35 up 19 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 12 KiB 18 GiB 2.2 TiB 75.43 1.06 39 up 22 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 20 KiB 17 GiB 2.8 TiB 69.57 0.98 38 up 25 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 1.0 MiB 21 GiB 1.1 TiB 88.03 1.24 50 up 30 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 96 KiB 19 GiB 1.6 TiB 82.62 1.16 48 up 33 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 64 KiB 18 GiB 2.2 TiB 75.97 1.07 36 up 34 hdd 9.09470 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 60 KiB 18 GiB 2.0 TiB 78.37 1.10 39 up 35 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 16 KiB 19 GiB 1.8 TiB 80.49 1.13 42 up 12 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 18 GiB 2.5 TiB 72.01 1.01 35 up 16 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 60 KiB 19 GiB 2.1 TiB 77.27 1.09 42 up 17 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 88 KiB 20 GiB 1.6 TiB 82.68 1.17 49 up 20 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 72 KiB 17 GiB 2.7 TiB 70.52 0.99 39 up 23 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 60 KiB 18 GiB 2.1 TiB 77.39 1.09 47 up 26 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 36 KiB 16 GiB 3.1 TiB 66.46 0.94 35 up 28 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 32 KiB 16 GiB 3.3 TiB 63.34 0.89 35 up 31 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 12 KiB 18 GiB 2.6 TiB 71.67 1.01 35 up 13 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 56 KiB 17 GiB 2.4 TiB 73.30 1.03 37 up 15 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 56 KiB 18 GiB 2.4 TiB 73.34 1.03 43 up 18 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 108 KiB 18 GiB 2.1 TiB 77.32 1.09 38 up 21 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 64 KiB 22 GiB 1.1 TiB 88.38 1.25 44 up 24 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 84 KiB 16 GiB 2.9 TiB 67.62 0.95 35 up 27 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 40 KiB 18 GiB 2.2 TiB 76.03 1.07 43 up 29 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 60 KiB 16 GiB 3.3 TiB 64.02 0.90 38 up 32 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 52 KiB 21 GiB 1.1 TiB 88.20 1.24 50 up 37 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.7 TiB 100 KiB 20 GiB 1.4 TiB 84.89 1.20 48 up 39 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 20 KiB 18 GiB 2.4 TiB 73.34 1.03 38 up 41 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 84 KiB 18 GiB 2.4 TiB 73.47 1.04 36 up 43 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 60 KiB 17 GiB 2.9 TiB 68.28 0.96 37 up 45 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 64 KiB 17 GiB 2.8 TiB 69.57 0.98 40 up 46 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 72 KiB 17 GiB 2.7 TiB 70.43 0.99 36 up 48 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 36 KiB 19 GiB 2.3 TiB 74.64 1.05 38 up 50 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 84 KiB 17 GiB 2.6 TiB 71.48 1.01 37 up 36 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 100 KiB 17 GiB 2.8 TiB 69.76 0.98 38 up 38 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 80 KiB 17 GiB 2.9 TiB 68.03 0.96 37 up 40 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 52 KiB 19 GiB 2.4 TiB 73.30 1.03 41 up 42 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 44 KiB 18 GiB 2.3 TiB 74.48 1.05 40 up 44 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 84 KiB 18 GiB 2.2 TiB 76.27 1.07 36 up 47 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 19 GiB 2.6 TiB 71.56 1.01 39 up 49 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 15 GiB 3.5 TiB 61.36 0.86 32 up 51 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 28 KiB 18 GiB 2.2 TiB 75.40 1.06 38 up 52 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 1.1 MiB 17 GiB 2.7 TiB 70.46 0.99 39 up 53 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 28 KiB 19 GiB 1.9 TiB 79.23 1.12 30 up 54 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 88 KiB 16 GiB 2.9 TiB 67.83 0.96 33 up 55 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 60 KiB 18 GiB 2.2 TiB 76.06 1.07 46 up 56 hdd 9.09470 1.00000 9.1 TiB 7.4 TiB 7.4 TiB 93 KiB 19 GiB 1.7 TiB 81.54 1.15 40 up 57 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 60 KiB 18 GiB 2.2 TiB 76.14 1.07 38 up 58 hdd 9.09470 1.00000 9.1 TiB 7.4 TiB 7.4 TiB 76 KiB 19 GiB 1.7 TiB 81.02 1.14 44 up 59 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 84 KiB 19 GiB 2.3 TiB 74.95 1.06 42 up 60 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 60 KiB 17 GiB 2.7 TiB 70.40 0.99 39 up 61 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 16 KiB 19 GiB 1.8 TiB 80.09 1.13 45 up 62 hdd 9.09470 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 96 KiB 18 GiB 2.0 TiB 78.39 1.10 46 up 63 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 136 KiB 18 GiB 2.5 TiB 73.03 1.03 41 up 64 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 36 KiB 18 GiB 2.1 TiB 77.16 1.09 39 up 65 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 144 KiB 18 GiB 2.2 TiB 75.83 1.07 41 up 66 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.7 TiB 104 KiB 20 GiB 1.4 TiB 84.43 1.19 41 up 67 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 108 KiB 19 GiB 1.8 TiB 80.46 1.13 43 up 84 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 76 KiB 18 GiB 2.2 TiB 75.61 1.07 42 up 85 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 72 KiB 19 GiB 1.9 TiB 79.25 1.12 43 up 86 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 32 KiB 19 GiB 1.8 TiB 80.40 1.13 45 up 87 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 48 KiB 18 GiB 2.5 TiB 72.34 1.02 40 up 88 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 32 KiB 19 GiB 1.8 TiB 79.95 1.13 45 up 89 hdd 9.09470 1.00000 9.1 TiB 7.4 TiB 7.3 TiB 40 KiB 20 GiB 1.7 TiB 81.02 1.14 44 up 90 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 28 KiB 17 GiB 3.0 TiB 66.48 0.94 36 up 91 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 96 KiB 16 GiB 2.9 TiB 67.92 0.96 37 up 76 hdd 9.09470 1.00000 9.1 TiB 7.4 TiB 7.4 TiB 48 KiB 19 GiB 1.7 TiB 81.26 1.14 42 up 77 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 17 GiB 3.1 TiB 66.28 0.93 37 up 78 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 89 KiB 17 GiB 2.8 TiB 69.66 0.98 39 up 79 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 32 KiB 17 GiB 2.8 TiB 69.22 0.98 40 up 80 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 76 KiB 18 GiB 2.5 TiB 72.68 1.02 36 up 81 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 112 KiB 16 GiB 2.9 TiB 67.93 0.96 35 up 82 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 64 KiB 18 GiB 2.3 TiB 74.52 1.05 38 up 83 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 7.9 TiB 84 KiB 21 GiB 1.1 TiB 87.43 1.23 45 up 68 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 24 KiB 18 GiB 2.6 TiB 71.01 1.00 36 up 69 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 20 KiB 20 GiB 1.9 TiB 79.06 1.11 38 up 70 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 60 KiB 18 GiB 2.4 TiB 73.13 1.03 39 up 71 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 32 KiB 19 GiB 2.1 TiB 76.58 1.08 43 up 72 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.5 TiB 36 KiB 17 GiB 2.5 TiB 72.04 1.02 39 up 73 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 56 KiB 16 GiB 3.0 TiB 67.55 0.95 32 up 74 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 8 KiB 18 GiB 2.4 TiB 73.25 1.03 42 up 75 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 52 KiB 17 GiB 3.1 TiB 65.90 0.93 33 up 121 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 72 KiB 18 GiB 2.7 TiB 70.71 1.00 39 up 123 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 88 KiB 20 GiB 1.1 TiB 87.65 1.23 49 up 124 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 76 KiB 17 GiB 2.9 TiB 68.20 0.96 35 up 126 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 56 KiB 18 GiB 2.0 TiB 77.51 1.09 46 up 128 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 KiB 18 GiB 2.2 TiB 75.69 1.07 40 up 130 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 104 KiB 18 GiB 1.8 TiB 79.71 1.12 37 up 132 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 28 KiB 19 GiB 1.6 TiB 82.35 1.16 42 up 134 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 16 KiB 17 GiB 3.2 TiB 65.11 0.92 36 up 112 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 52 KiB 19 GiB 2.2 TiB 75.60 1.07 40 up 113 hdd 9.09470 1.00000 9.1 TiB 7.4 TiB 7.3 TiB 88 KiB 19 GiB 1.7 TiB 81.00 1.14 43 up 114 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 36 KiB 18 GiB 2.4 TiB 73.71 1.04 43 up 115 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 32 KiB 20 GiB 1.5 TiB 83.99 1.18 43 up 116 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.4 TiB 92 KiB 19 GiB 1.6 TiB 81.97 1.15 42 up 117 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.1 TiB 68 KiB 19 GiB 1.9 TiB 78.63 1.11 44 up 118 hdd 9.09470 1.00000 9.1 TiB 7.9 TiB 7.9 TiB 128 KiB 20 GiB 1.2 TiB 87.25 1.23 40 up 119 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 KiB 18 GiB 2.1 TiB 76.39 1.08 37 up 120 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 52 KiB 17 GiB 2.6 TiB 71.55 1.01 39 up 122 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 28 KiB 18 GiB 2.1 TiB 76.36 1.08 37 up 125 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 64 KiB 19 GiB 2.0 TiB 77.49 1.09 42 up 127 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.5 TiB 1.1 MiB 18 GiB 2.5 TiB 72.14 1.02 44 up 129 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 64 KiB 19 GiB 1.9 TiB 78.86 1.11 45 up 131 hdd 9.09470 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 44 KiB 19 GiB 2.0 TiB 78.55 1.11 43 up 133 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 36 KiB 18 GiB 2.5 TiB 73.01 1.03 38 up 135 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 112 KiB 18 GiB 2.2 TiB 76.06 1.07 39 up 92 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 56 KiB 17 GiB 2.4 TiB 73.87 1.04 41 up 93 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 32 KiB 18 GiB 2.4 TiB 73.95 1.04 43 up 94 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 16 GiB 3.1 TiB 66.46 0.94 35 up 95 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 72 KiB 16 GiB 2.9 TiB 67.58 0.95 35 up 96 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 92 KiB 17 GiB 2.9 TiB 68.22 0.96 37 up 97 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 92 KiB 18 GiB 2.1 TiB 77.41 1.09 39 up 98 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 36 KiB 19 GiB 1.9 TiB 79.51 1.12 37 up 99 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 32 KiB 17 GiB 2.4 TiB 73.31 1.03 41 up
Which means if someone does a bigger rebalance or PG raise there's a risk of OSDs running full causing pool shutdown.
This NEVER happend under Luminous.
#3 Updated by Anonymous over 4 years ago
This happens with active PG balancer if the cluster is in WARN state.
...
51 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 100 KiB 15 GiB 3.3 TiB 64.01 0.98 24 up
52 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 3.1 MiB 16 GiB 2.8 TiB 69.62 1.07 28 up
53 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 32 KiB 21 GiB 1.0 TiB 88.47 1.36 31 up
54 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 100 KiB 17 GiB 2.4 TiB 73.44 1.13 28 up
55 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.5 TiB 40 KiB 15 GiB 3.6 TiB 60.27 0.93 27 up
...
Major mismatch here.
I posted about this so many times now, I replicated it, nobody seems to bother. It's a MAJOR issue.
#4 Updated by Anonymous over 4 years ago
ceph balancer status
{
"active": true,
"plans": [],
"mode": "upmap"
}
bad distribution:
0 ssd 3.49219 1.00000 3.5 TiB 713 GiB 673 GiB 36 GiB 3.6 GiB 2.8 TiB 19.93 0.29 147 up 1 ssd 3.49219 1.00000 3.5 TiB 724 GiB 671 GiB 49 GiB 3.9 GiB 2.8 TiB 20.24 0.30 146 up 2 ssd 3.49219 1.00000 3.5 TiB 735 GiB 680 GiB 50 GiB 4.2 GiB 2.8 TiB 20.54 0.30 150 up 3 ssd 3.49219 1.00000 3.5 TiB 712 GiB 675 GiB 33 GiB 3.5 GiB 2.8 TiB 19.91 0.29 146 up 4 ssd 3.49219 1.00000 3.5 TiB 752 GiB 713 GiB 34 GiB 4.8 GiB 2.8 TiB 21.02 0.31 156 up 6 ssd 3.49219 1.00000 3.5 TiB 709 GiB 670 GiB 35 GiB 3.8 GiB 2.8 TiB 19.83 0.29 146 up 8 ssd 3.49219 1.00000 3.5 TiB 781 GiB 737 GiB 40 GiB 3.7 GiB 2.7 TiB 21.83 0.32 156 up 10 ssd 3.49219 1.00000 3.5 TiB 727 GiB 681 GiB 42 GiB 3.8 GiB 2.8 TiB 20.33 0.30 146 up 5 ssd 3.49219 1.00000 3.5 TiB 664 GiB 628 GiB 32 GiB 4.3 GiB 2.8 TiB 18.57 0.27 141 up 7 ssd 3.49219 1.00000 3.5 TiB 656 GiB 613 GiB 39 GiB 4.1 GiB 2.9 TiB 18.35 0.27 136 up 9 ssd 3.49219 1.00000 3.5 TiB 631 GiB 585 GiB 41 GiB 4.0 GiB 2.9 TiB 17.64 0.26 131 up 11 ssd 3.49219 1.00000 3.5 TiB 725 GiB 701 GiB 22 GiB 2.4 GiB 2.8 TiB 20.26 0.30 138 up 101 ssd 3.49219 1.00000 3.5 TiB 754 GiB 712 GiB 38 GiB 4.0 GiB 2.8 TiB 21.09 0.31 146 up 103 ssd 3.49219 1.00000 3.5 TiB 761 GiB 717 GiB 40 GiB 3.5 GiB 2.7 TiB 21.28 0.31 150 up 105 ssd 3.49219 1.00000 3.5 TiB 715 GiB 676 GiB 36 GiB 2.8 GiB 2.8 TiB 19.98 0.29 148 up 107 ssd 3.49219 1.00000 3.5 TiB 759 GiB 706 GiB 50 GiB 3.2 GiB 2.8 TiB 21.23 0.31 147 up 100 ssd 3.49219 1.00000 3.5 TiB 724 GiB 674 GiB 47 GiB 2.4 GiB 2.8 TiB 20.24 0.30 144 up 102 ssd 3.49219 1.00000 3.5 TiB 669 GiB 654 GiB 12 GiB 2.5 GiB 2.8 TiB 18.70 0.27 141 up 104 ssd 3.49219 1.00000 3.5 TiB 720 GiB 687 GiB 31 GiB 3.0 GiB 2.8 TiB 20.14 0.30 144 up 106 ssd 3.49219 1.00000 3.5 TiB 713 GiB 645 GiB 65 GiB 3.2 GiB 2.8 TiB 19.95 0.29 143 up 108 ssd 3.49219 1.00000 3.5 TiB 728 GiB 690 GiB 36 GiB 2.6 GiB 2.8 TiB 20.36 0.30 156 up 109 ssd 3.49219 1.00000 3.5 TiB 731 GiB 684 GiB 45 GiB 3.0 GiB 2.8 TiB 20.46 0.30 146 up 110 ssd 3.49219 1.00000 3.5 TiB 773 GiB 742 GiB 28 GiB 2.6 GiB 2.7 TiB 21.62 0.32 154 up 111 ssd 3.49219 1.00000 3.5 TiB 707 GiB 659 GiB 45 GiB 2.7 GiB 2.8 TiB 19.76 0.29 146 up
#5 Updated by Anonymous about 4 years ago
After being told on the ML that my low PG count is supposed to be the reason, I raised it. Nothing changed, still same issues here.
14 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 96 KiB 21 GiB 1.1 TiB 88.17 1.34 159 up 19 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 72 KiB 17 GiB 2.8 TiB 69.29 1.05 126 up 22 hdd 9.09470 1.00000 9.1 TiB 7.9 TiB 7.9 TiB 132 KiB 21 GiB 1.2 TiB 87.15 1.33 156 up 25 hdd 9.09470 1.00000 9.1 TiB 7.3 TiB 7.3 TiB 6.4 MiB 19 GiB 1.8 TiB 80.14 1.22 140 up 30 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 88 KiB 17 GiB 2.4 TiB 73.39 1.12 130 up 33 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 36 KiB 18 GiB 2.3 TiB 74.47 1.13 131 up 34 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 96 KiB 19 GiB 1.9 TiB 79.18 1.20 142 up 35 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 92 KiB 17 GiB 2.7 TiB 70.64 1.07 129 up 12 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 56 KiB 17 GiB 2.5 TiB 72.25 1.10 129 up 16 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 76 KiB 18 GiB 2.2 TiB 76.22 1.16 137 up 17 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 112 KiB 18 GiB 2.1 TiB 77.11 1.17 142 up 20 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 72 KiB 17 GiB 2.8 TiB 69.42 1.06 125 up 23 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 56 KiB 18 GiB 2.5 TiB 72.35 1.10 131 up 26 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 80 KiB 17 GiB 2.8 TiB 69.50 1.06 126 up 28 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 116 KiB 17 GiB 3.2 TiB 64.59 0.98 119 up 31 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 56 KiB 16 GiB 3.3 TiB 63.28 0.96 114 up 13 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 44 KiB 18 GiB 2.8 TiB 69.23 1.05 126 up 15 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 16 GiB 3.1 TiB 66.27 1.01 121 up 18 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 8 KiB 17 GiB 2.8 TiB 69.64 1.06 124 up 21 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 132 KiB 18 GiB 2.1 TiB 76.81 1.17 142 up 24 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 16 GiB 3.2 TiB 65.08 0.99 118 up 27 hdd 9.09470 1.00000 9.1 TiB 7.2 TiB 7.2 TiB 12 KiB 19 GiB 1.9 TiB 78.82 1.20 144 up 29 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 108 KiB 16 GiB 3.2 TiB 65.22 0.99 117 up 32 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 124 KiB 19 GiB 1.6 TiB 82.74 1.26 150 up 37 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 56 KiB 16 GiB 3.2 TiB 65.34 0.99 118 up 39 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 44 KiB 18 GiB 2.2 TiB 76.05 1.16 134 up 41 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB 16 GiB 3.4 TiB 62.90 0.96 113 up 43 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 32 KiB 18 GiB 2.4 TiB 73.15 1.11 129 up 45 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB 16 GiB 3.4 TiB 62.72 0.95 113 up 46 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 136 KiB 17 GiB 2.9 TiB 68.46 1.04 124 up 48 hdd 9.09470 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 56 KiB 15 GiB 3.9 TiB 57.64 0.88 104 up 50 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB 16 GiB 3.0 TiB 67.37 1.02 120 up 36 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 64 KiB 18 GiB 2.3 TiB 74.60 1.13 134 up 38 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 120 KiB 18 GiB 2.4 TiB 73.83 1.12 131 up 40 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 48 KiB 18 GiB 2.5 TiB 72.69 1.11 131 up 42 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 18 GiB 2.5 TiB 72.01 1.10 129 up 44 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.4 TiB 104 KiB 15 GiB 3.6 TiB 60.06 0.91 109 up 47 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 48 KiB 17 GiB 2.6 TiB 70.99 1.08 130 up 49 hdd 9.09470 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 92 KiB 15 GiB 3.9 TiB 57.54 0.88 102 up 51 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 72 KiB 16 GiB 3.2 TiB 64.81 0.99 116 up 52 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 6.3 MiB 18 GiB 2.3 TiB 74.18 1.13 134 up 53 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 104 KiB 16 GiB 3.2 TiB 64.99 0.99 114 up 54 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.4 TiB 52 KiB 15 GiB 3.7 TiB 59.57 0.91 108 up 55 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 52 KiB 19 GiB 2.1 TiB 76.56 1.16 136 up 56 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 72 KiB 18 GiB 2.6 TiB 71.86 1.09 125 up 57 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB 16 GiB 3.3 TiB 63.59 0.97 115 up 58 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB 15 GiB 3.5 TiB 61.75 0.94 110 up 59 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 60 KiB 18 GiB 2.4 TiB 73.20 1.11 132 up 60 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 60 KiB 16 GiB 3.2 TiB 65.00 0.99 116 up 61 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 112 KiB 16 GiB 3.1 TiB 65.86 1.00 117 up 62 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 120 KiB 18 GiB 2.6 TiB 71.59 1.09 127 up 63 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 16 KiB 17 GiB 2.9 TiB 68.06 1.04 120 up 64 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.7 TiB 96 KiB 18 GiB 2.3 TiB 74.30 1.13 133 up 65 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 64 KiB 18 GiB 2.6 TiB 71.52 1.09 127 up 66 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 52 KiB 15 GiB 3.5 TiB 61.53 0.94 107 up 67 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.3 TiB 52 KiB 15 GiB 3.7 TiB 58.88 0.90 105 up 84 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 16 GiB 3.3 TiB 63.58 0.97 114 up 85 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 20 KiB 19 GiB 2.2 TiB 75.81 1.15 135 up 86 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 92 KiB 18 GiB 2.7 TiB 70.31 1.07 128 up 87 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 44 KiB 18 GiB 2.4 TiB 73.16 1.11 130 up 88 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 64 KiB 17 GiB 2.8 TiB 69.07 1.05 125 up 89 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB 16 GiB 3.0 TiB 66.83 1.02 122 up 90 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 84 KiB 17 GiB 2.9 TiB 68.15 1.04 120 up 91 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.3 TiB 48 KiB 15 GiB 3.7 TiB 58.97 0.90 104 up 76 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 100 KiB 16 GiB 3.3 TiB 63.62 0.97 115 up 77 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 16 GiB 3.4 TiB 62.08 0.94 110 up 78 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 16 GiB 3.3 TiB 64.12 0.98 115 up 79 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 84 KiB 16 GiB 3.3 TiB 63.57 0.97 115 up 80 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 68 KiB 19 GiB 2.1 TiB 76.64 1.17 138 up 81 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 49 KiB 16 GiB 2.9 TiB 67.64 1.03 121 up 82 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 36 KiB 17 GiB 2.9 TiB 68.03 1.03 120 up 83 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 68 KiB 17 GiB 2.8 TiB 69.21 1.05 121 up 68 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 32 KiB 18 GiB 2.3 TiB 74.22 1.13 133 up 69 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 72 KiB 17 GiB 2.8 TiB 68.81 1.05 123 up 70 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 36 KiB 17 GiB 2.9 TiB 68.15 1.04 123 up 71 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 128 KiB 16 GiB 3.2 TiB 64.29 0.98 114 up 72 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 76 KiB 16 GiB 3.1 TiB 65.56 1.00 120 up 73 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 16 GiB 3.3 TiB 64.18 0.98 118 up 74 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 92 KiB 17 GiB 2.6 TiB 71.68 1.09 131 up 75 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB 15 GiB 3.5 TiB 61.78 0.94 112 up 180 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 16 GiB 3.2 TiB 64.73 0.98 115 up 186 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB 16 GiB 2.9 TiB 68.51 1.04 124 up 192 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 72 KiB 17 GiB 2.9 TiB 68.41 1.04 124 up 197 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 56 KiB 15 GiB 3.3 TiB 63.70 0.97 114 up 202 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 104 KiB 16 GiB 3.2 TiB 65.07 0.99 116 up 208 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 32 KiB 18 GiB 2.2 TiB 75.45 1.15 133 up 213 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 6.3 MiB 16 GiB 3.3 TiB 64.02 0.97 117 up 219 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 92 KiB 15 GiB 3.2 TiB 65.16 0.99 118 up 176 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 88 KiB 16 GiB 3.3 TiB 63.89 0.97 115 up 182 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 84 KiB 17 GiB 2.8 TiB 69.52 1.06 127 up 188 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 64 KiB 17 GiB 2.6 TiB 71.13 1.08 127 up 196 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 48 KiB 17 GiB 2.9 TiB 68.14 1.04 122 up 201 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 52 KiB 15 GiB 3.4 TiB 62.63 0.95 114 up 207 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 104 KiB 17 GiB 2.6 TiB 71.05 1.08 131 up 214 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB 16 GiB 2.8 TiB 69.53 1.06 125 up 218 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 60 KiB 16 GiB 3.2 TiB 64.81 0.99 119 up 179 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 84 KiB 15 GiB 3.1 TiB 65.64 1.00 119 up 185 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 132 KiB 16 GiB 3.0 TiB 66.54 1.01 119 up 191 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB 17 GiB 2.9 TiB 68.46 1.04 122 up 198 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB 3.1 TiB 65.92 1.00 121 up 203 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 108 KiB 18 GiB 2.2 TiB 75.79 1.15 134 up 209 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 36 KiB 17 GiB 3.1 TiB 65.78 1.00 120 up 215 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 52 KiB 16 GiB 3.1 TiB 65.68 1.00 120 up 221 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 68 KiB 15 GiB 3.4 TiB 62.19 0.95 111 up 177 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 18 GiB 2.6 TiB 71.44 1.09 130 up 183 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 88 KiB 18 GiB 2.2 TiB 76.23 1.16 141 up 190 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 52 KiB 18 GiB 2.3 TiB 74.20 1.13 133 up 194 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB 3.1 TiB 65.70 1.00 122 up 199 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 104 KiB 17 GiB 2.7 TiB 70.36 1.07 122 up 205 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 16 GiB 3.1 TiB 66.22 1.01 120 up 211 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 88 KiB 17 GiB 2.3 TiB 74.83 1.14 137 up 217 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 17 GiB 2.6 TiB 71.71 1.09 130 up 178 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB 15 GiB 3.4 TiB 62.70 0.95 115 up 184 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 16 GiB 3.2 TiB 64.77 0.99 119 up 189 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 92 KiB 13 GiB 4.2 TiB 54.30 0.83 98 up 195 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 36 KiB 16 GiB 3.1 TiB 66.27 1.01 118 up 200 hdd 9.09470 1.00000 9.1 TiB 5.3 TiB 5.3 TiB 56 KiB 15 GiB 3.8 TiB 57.95 0.88 103 up 206 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.5 TiB 80 KiB 15 GiB 3.6 TiB 60.59 0.92 109 up 212 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 132 KiB 16 GiB 2.9 TiB 67.71 1.03 124 up 216 hdd 9.09470 1.00000 9.1 TiB 5.1 TiB 5.1 TiB 56 KiB 14 GiB 4.0 TiB 56.28 0.86 99 up 136 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 15 GiB 3.3 TiB 64.08 0.97 114 up 137 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 52 KiB 16 GiB 3.2 TiB 65.23 0.99 118 up 138 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.5 TiB 92 KiB 17 GiB 2.5 TiB 72.17 1.10 132 up 139 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 56 KiB 17 GiB 2.8 TiB 69.18 1.05 125 up 140 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 108 KiB 16 GiB 3.2 TiB 64.86 0.99 115 up 141 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 44 KiB 16 GiB 3.0 TiB 66.72 1.02 121 up 142 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 88 KiB 18 GiB 2.2 TiB 76.21 1.16 138 up 143 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 100 KiB 16 GiB 3.2 TiB 64.96 0.99 116 up 121 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 32 KiB 17 GiB 2.8 TiB 68.97 1.05 121 up 123 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 136 KiB 19 GiB 2.4 TiB 74.10 1.13 131 up 124 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 128 KiB 16 GiB 3.0 TiB 67.48 1.03 121 up 126 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 20 KiB 18 GiB 2.8 TiB 69.55 1.06 125 up 128 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 96 KiB 16 GiB 3.0 TiB 66.99 1.02 120 up 130 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 24 KiB 18 GiB 2.9 TiB 68.49 1.04 125 up 132 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 40 KiB 17 GiB 2.9 TiB 68.11 1.04 122 up 134 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 80 KiB 16 GiB 3.5 TiB 61.26 0.93 106 up 146 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 64 KiB 16 GiB 2.8 TiB 68.86 1.05 124 up 149 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 96 KiB 18 GiB 2.4 TiB 73.83 1.12 129 up 152 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 28 KiB 17 GiB 2.8 TiB 68.94 1.05 124 up 155 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB 2.7 TiB 70.56 1.07 128 up 158 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 52 KiB 17 GiB 2.6 TiB 70.92 1.08 129 up 161 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 56 KiB 16 GiB 3.0 TiB 66.87 1.02 122 up 164 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 52 KiB 17 GiB 2.6 TiB 71.05 1.08 127 up 167 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 108 KiB 15 GiB 3.4 TiB 63.07 0.96 117 up 144 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB 16 GiB 3.0 TiB 67.00 1.02 120 up 147 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 93 KiB 16 GiB 2.8 TiB 69.41 1.06 123 up 150 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 96 KiB 16 GiB 2.8 TiB 69.12 1.05 125 up 153 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 40 KiB 17 GiB 2.8 TiB 68.67 1.04 125 up 156 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 48 KiB 18 GiB 2.7 TiB 70.15 1.07 128 up 159 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 60 KiB 16 GiB 3.0 TiB 67.03 1.02 119 up 163 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 56 KiB 18 GiB 2.1 TiB 76.58 1.16 138 up 166 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 88 KiB 16 GiB 3.1 TiB 65.72 1.00 120 up 145 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 92 KiB 16 GiB 2.7 TiB 70.00 1.06 127 up 148 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 124 KiB 16 GiB 2.9 TiB 68.34 1.04 124 up 151 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 80 KiB 16 GiB 3.0 TiB 67.09 1.02 118 up 154 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 40 KiB 16 GiB 3.0 TiB 66.74 1.02 121 up 157 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 92 KiB 17 GiB 3.0 TiB 66.90 1.02 116 up 160 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 84 KiB 17 GiB 2.9 TiB 68.53 1.04 125 up 162 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB 2.6 TiB 71.80 1.09 129 up 165 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 48 KiB 17 GiB 2.4 TiB 73.32 1.12 131 up 112 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 80 KiB 17 GiB 2.9 TiB 68.16 1.04 123 up 113 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 88 KiB 16 GiB 3.2 TiB 65.04 0.99 115 up 114 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 64 KiB 16 GiB 3.2 TiB 64.73 0.98 117 up 115 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 72 KiB 17 GiB 3.0 TiB 67.26 1.02 119 up 116 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 24 KiB 16 GiB 3.3 TiB 63.48 0.97 115 up 117 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.8 TiB 84 KiB 19 GiB 2.2 TiB 75.42 1.15 138 up 118 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 92 KiB 17 GiB 2.8 TiB 69.45 1.06 124 up 119 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 15 GiB 3.5 TiB 61.80 0.94 112 up 120 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 92 KiB 16 GiB 3.3 TiB 63.40 0.96 116 up 122 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 52 KiB 16 GiB 3.0 TiB 66.59 1.01 118 up 125 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB 16 GiB 3.4 TiB 62.73 0.95 114 up 127 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 18 GiB 2.6 TiB 71.58 1.09 129 up 129 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB 2.7 TiB 70.06 1.07 125 up 131 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.4 TiB 108 KiB 15 GiB 3.6 TiB 59.98 0.91 108 up 133 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 15 GiB 3.5 TiB 61.28 0.93 109 up 135 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 15 GiB 3.5 TiB 61.80 0.94 113 up 168 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB 16 GiB 2.9 TiB 67.94 1.03 124 up 169 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 80 KiB 17 GiB 2.8 TiB 69.43 1.06 125 up 170 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 48 KiB 15 GiB 3.4 TiB 62.67 0.95 111 up 171 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 44 KiB 15 GiB 3.2 TiB 65.12 0.99 119 up 172 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 32 KiB 16 GiB 3.0 TiB 67.19 1.02 121 up 173 hdd 9.09470 1.00000 9.1 TiB 5.4 TiB 5.4 TiB 84 KiB 14 GiB 3.7 TiB 59.32 0.90 104 up 174 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.5 TiB 56 KiB 15 GiB 3.6 TiB 60.40 0.92 109 up 175 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 84 KiB 17 GiB 2.7 TiB 70.61 1.07 121 up 181 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 44 KiB 18 GiB 2.4 TiB 73.69 1.12 134 up 187 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 88 KiB 16 GiB 2.8 TiB 69.02 1.05 122 up 193 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 64 KiB 17 GiB 2.5 TiB 72.77 1.11 133 up 204 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 52 KiB 16 GiB 2.9 TiB 68.53 1.04 124 up 210 hdd 9.09470 1.00000 9.1 TiB 6.9 TiB 6.9 TiB 100 KiB 18 GiB 2.2 TiB 76.29 1.16 138 up 220 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 17 GiB 2.6 TiB 71.89 1.09 133 up 222 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 96 KiB 17 GiB 2.5 TiB 72.35 1.10 132 up 223 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB 2.5 TiB 72.55 1.10 131 up 92 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 72 KiB 17 GiB 2.9 TiB 67.94 1.03 123 up 93 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 84 KiB 17 GiB 2.6 TiB 71.01 1.08 129 up 94 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 84 KiB 17 GiB 2.8 TiB 69.07 1.05 123 up 95 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 16 GiB 3.1 TiB 65.81 1.00 120 up 96 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 80 KiB 16 GiB 2.9 TiB 68.07 1.04 121 up 97 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 52 KiB 17 GiB 2.8 TiB 69.25 1.05 127 up 98 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 124 KiB 17 GiB 2.8 TiB 69.55 1.06 126 up 99 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 84 KiB 16 GiB 3.1 TiB 66.04 1.00 120 up
There's 3 OSD with > 80% and one at almost 90 whilst others are in the low 60s.
Upmade PG Balancer mode. Someone on the ML said there are fixes introduced soon? There is some major bug there
#6 Updated by Anonymous about 4 years ago
... 19 hdd 9.09470 1.00000 9.1 TiB 6.8 TiB 6.8 TiB 92 KiB 18 GiB 2.2 TiB 75.28 1.07 132 up 22 hdd 9.09470 1.00000 9.1 TiB 7.1 TiB 7.1 TiB 88 KiB 19 GiB 2.0 TiB 78.20 1.11 137 up 25 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 11 MiB 21 GiB 1.1 TiB 87.80 1.25 155 up 30 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 60 KiB 18 GiB 2.3 TiB 74.20 1.05 130 up 33 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 80 KiB 18 GiB 2.4 TiB 74.12 1.05 130 up ...
OSD 25 nearfull, so I reweight to 0.9 and what happens?
25 hdd 9.09470 0.89999 9.1 TiB 8.0 TiB 8.0 TiB 11 MiB 21 GiB 1.1 TiB 87.87 1.25 155 up
It's moving data ONTO the OSD, not OFF.