Actions
Bug #49197
closedpg balancer not working during recovery --> bad distribution --> full pools --> complete cluster failure
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
It seems that PG balacner in upmap mode does not work during recovery. Last night we lost a HDD and during the night writes on the SSD pool were heavy and not distributed properly causing complete read-only failure this morning.
14 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 132 KiB 29 GiB 3.4 TiB 62.18 0.99 102 up 19 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 148 KiB 29 GiB 3.4 TiB 62.87 1.00 102 up 22 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 108 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 25 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 13 MiB 29 GiB 3.4 TiB 62.46 1.00 102 up 30 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 44 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 33 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 96 KiB 29 GiB 3.4 TiB 62.37 0.99 101 up 34 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 56 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 35 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 52 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 233 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 236 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 160 KiB 29 GiB 3.4 TiB 62.16 0.99 102 up 239 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 52 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 242 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 68 KiB 28 GiB 3.5 TiB 61.71 0.98 101 up 244 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 20 KiB 29 GiB 3.4 TiB 62.94 1.00 102 up 247 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 84 KiB 29 GiB 3.4 TiB 62.37 0.99 101 up 250 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 24 KiB 29 GiB 3.5 TiB 61.74 0.98 101 up 253 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 168 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 120 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 24 KiB 29 GiB 3.4 TiB 62.85 1.00 102 up 122 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 20 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 125 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 28 GiB 3.5 TiB 61.76 0.98 101 up 127 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 12 KiB 29 GiB 3.5 TiB 61.75 0.98 101 up 129 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 68 KiB 29 GiB 3.4 TiB 62.86 1.00 102 up 131 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 84 KiB 28 GiB 3.4 TiB 62.20 0.99 102 up 133 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 135 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 96 KiB 28 GiB 3.4 TiB 62.19 0.99 102 up 234 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 29 GiB 3.5 TiB 61.75 0.98 101 up 238 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB 29 GiB 3.4 TiB 62.88 1.00 102 up 241 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 116 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 245 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 156 KiB 28 GiB 3.4 TiB 62.15 0.99 102 up 249 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 60 KiB 29 GiB 3.5 TiB 61.76 0.98 101 up 252 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 108 KiB 29 GiB 3.5 TiB 61.33 0.98 100 up 254 hdd 9.09569 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 88 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 255 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 88 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 232 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 52 KiB 29 GiB 3.4 TiB 62.87 1.00 102 up 235 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 92 KiB 29 GiB 3.4 TiB 62.79 1.00 102 up 237 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 60 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 240 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 243 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 84 KiB 29 GiB 3.4 TiB 62.50 1.00 101 up 246 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 8 KiB 29 GiB 3.4 TiB 62.82 1.00 102 up 248 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 76 KiB 29 GiB 3.4 TiB 62.81 1.00 102 up 251 hdd 9.09569 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 156 KiB 29 GiB 3.3 TiB 63.28 1.01 103 up 12 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 16 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 80 KiB 29 GiB 3.4 TiB 62.86 1.00 102 up 17 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 116 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 20 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 92 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 23 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 16 KiB 28 GiB 3.5 TiB 61.27 0.98 100 up 26 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 44 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 28 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 68 KiB 28 GiB 3.5 TiB 61.35 0.98 100 up 31 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 60 KiB 29 GiB 3.4 TiB 62.40 0.99 101 up 13 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 28 GiB 3.5 TiB 61.73 0.98 101 up 15 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 56 KiB 29 GiB 3.5 TiB 61.33 0.98 100 up 18 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 68 KiB 28 GiB 3.5 TiB 61.73 0.98 101 up 21 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 76 KiB 28 GiB 3.5 TiB 61.35 0.98 100 up 24 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 80 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 27 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 28 KiB 28 GiB 3.5 TiB 61.60 0.98 100 up 29 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 32 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 88 KiB 29 GiB 3.4 TiB 62.48 1.00 101 up 37 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 72 KiB 29 GiB 3.4 TiB 62.82 1.00 102 up 39 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 44 KiB 29 GiB 3.3 TiB 63.29 1.01 103 up 41 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 72 KiB 29 GiB 3.4 TiB 62.44 0.99 101 up 43 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 24 KiB 29 GiB 3.4 TiB 62.79 1.00 102 up 45 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 72 KiB 29 GiB 3.4 TiB 62.86 1.00 102 up 46 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 64 KiB 29 GiB 3.4 TiB 62.40 0.99 101 up 48 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 28 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 50 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 68 KiB 29 GiB 3.4 TiB 62.39 0.99 101 up 36 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 128 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 38 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 68 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 40 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 96 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 42 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 52 KiB 29 GiB 3.4 TiB 62.87 1.00 102 up 44 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 68 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 47 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 28 GiB 3.5 TiB 61.35 0.98 100 up 49 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 51 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 20 KiB 28 GiB 3.5 TiB 61.27 0.98 100 up 52 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 13 MiB 31 GiB 2.9 TiB 68.22 1.09 106 up 53 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 168 KiB 32 GiB 2.9 TiB 68.08 1.08 101 up 54 hdd 9.09470 0 0 B 0 B 0 B 0 B 0 B 0 B 0 0 0 down 55 hdd 9.09569 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 60 KiB 31 GiB 3.0 TiB 66.82 1.06 105 up 56 hdd 9.09470 1.00000 9.1 TiB 5.9 TiB 5.8 TiB 228 KiB 30 GiB 3.2 TiB 64.38 1.03 102 up 57 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 332 KiB 31 GiB 3.0 TiB 67.41 1.07 101 up 58 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 176 KiB 29 GiB 3.3 TiB 63.54 1.01 102 up 59 hdd 9.09569 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 316 KiB 29 GiB 3.2 TiB 64.79 1.03 105 up 60 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 76 KiB 30 GiB 3.4 TiB 62.43 0.99 101 up 61 hdd 9.09569 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 64 KiB 28 GiB 3.3 TiB 63.25 1.01 103 up 62 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 40 KiB 30 GiB 3.4 TiB 62.46 1.00 101 up 63 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 72 KiB 29 GiB 3.4 TiB 62.39 0.99 101 up 64 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 76 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 65 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 100 KiB 29 GiB 3.4 TiB 62.36 0.99 101 up 66 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 44 KiB 30 GiB 3.4 TiB 62.43 0.99 101 up 67 hdd 9.09569 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 216 KiB 31 GiB 3.3 TiB 63.69 1.01 104 up 84 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 60 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 85 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 28 KiB 29 GiB 3.5 TiB 61.33 0.98 100 up 86 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 96 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 87 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 92 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 88 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 28 KiB 29 GiB 3.4 TiB 62.40 0.99 101 up 89 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 36 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 90 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 80 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 91 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 48 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 76 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 68 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 77 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 100 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 78 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 136 KiB 28 GiB 3.5 TiB 61.30 0.98 100 up 79 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 29 GiB 3.5 TiB 61.38 0.98 100 up 80 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 40 KiB 30 GiB 3.4 TiB 62.40 0.99 101 up 81 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 92 KiB 29 GiB 3.4 TiB 62.45 0.99 101 up 82 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 28 KiB 29 GiB 3.5 TiB 61.37 0.98 100 up 83 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 68 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 80 KiB 29 GiB 3.4 TiB 62.84 1.00 102 up 69 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 40 KiB 29 GiB 3.5 TiB 61.27 0.98 100 up 70 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 40 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 71 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 80 KiB 29 GiB 3.4 TiB 62.44 0.99 101 up 72 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 28 GiB 3.5 TiB 61.82 0.98 101 up 73 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 32 KiB 28 GiB 3.5 TiB 61.27 0.98 100 up 74 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 104 KiB 29 GiB 3.4 TiB 62.44 0.99 101 up 75 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 56 KiB 29 GiB 3.4 TiB 62.41 0.99 101 up 180 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 44 KiB 29 GiB 3.4 TiB 62.47 1.00 101 up 186 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 40 KiB 28 GiB 3.5 TiB 61.76 0.98 101 up 192 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 28 GiB 3.5 TiB 61.71 0.98 101 up 197 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 36 KiB 28 GiB 3.5 TiB 61.78 0.98 101 up 202 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 64 KiB 29 GiB 3.4 TiB 62.38 0.99 101 up 208 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 80 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 213 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 13 MiB 28 GiB 3.5 TiB 61.74 0.98 102 up 219 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 96 KiB 28 GiB 3.4 TiB 62.16 0.99 102 up 176 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 64 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 182 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB 29 GiB 3.4 TiB 62.82 1.00 102 up 188 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB 28 GiB 3.5 TiB 61.29 0.98 100 up 196 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 48 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 201 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 28 GiB 3.5 TiB 61.38 0.98 100 up 207 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 56 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 214 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 92 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 218 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 60 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 179 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 76 KiB 28 GiB 3.5 TiB 61.72 0.98 101 up 185 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 128 KiB 29 GiB 3.4 TiB 62.46 1.00 101 up 191 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 198 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 44 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 203 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 104 KiB 28 GiB 3.5 TiB 61.75 0.98 101 up 209 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 120 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 215 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 60 KiB 28 GiB 3.5 TiB 61.81 0.98 101 up 221 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 88 KiB 28 GiB 3.5 TiB 61.36 0.98 100 up 177 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 80 KiB 29 GiB 3.4 TiB 62.85 1.00 102 up 183 hdd 9.09569 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 112 KiB 29 GiB 3.4 TiB 62.81 1.00 102 up 190 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 148 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 194 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 60 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 199 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 192 KiB 29 GiB 3.4 TiB 62.87 1.00 102 up 205 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 211 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 28 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 217 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 80 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 178 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 40 KiB 28 GiB 3.5 TiB 61.76 0.98 101 up 184 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 60 KiB 28 GiB 3.4 TiB 62.35 0.99 101 up 189 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 56 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 195 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 32 KiB 28 GiB 3.5 TiB 61.18 0.97 99 up 200 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 80 KiB 29 GiB 3.4 TiB 62.39 0.99 101 up 206 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 184 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 212 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 116 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 216 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 28 GiB 3.5 TiB 61.79 0.98 101 up 136 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 92 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 137 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 64 KiB 29 GiB 3.4 TiB 62.47 1.00 101 up 138 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 52 KiB 29 GiB 3.4 TiB 62.41 0.99 101 up 139 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 60 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 140 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 36 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 141 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 52 KiB 28 GiB 3.5 TiB 61.30 0.98 100 up 142 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 52 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 143 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 104 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 121 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 48 KiB 30 GiB 3.4 TiB 62.37 0.99 101 up 123 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.7 TiB 36 KiB 29 GiB 3.3 TiB 63.34 1.01 103 up 124 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 44 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 126 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB 29 GiB 3.4 TiB 62.88 1.00 102 up 128 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 124 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 130 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 80 KiB 29 GiB 3.4 TiB 62.41 0.99 101 up 132 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 96 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 134 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 61 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 146 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 200 KiB 28 GiB 3.5 TiB 61.75 0.98 101 up 149 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 152 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 88 KiB 28 GiB 3.4 TiB 62.27 0.99 102 up 155 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 40 KiB 29 GiB 3.4 TiB 62.44 0.99 101 up 158 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 108 KiB 28 GiB 3.5 TiB 61.71 0.98 101 up 161 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 52 KiB 28 GiB 3.5 TiB 61.75 0.98 101 up 164 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 59 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 167 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 80 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 144 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 76 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 147 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 168 KiB 28 GiB 3.5 TiB 61.35 0.98 100 up 150 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 108 KiB 29 GiB 3.4 TiB 62.41 0.99 101 up 153 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 28 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 156 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 92 KiB 28 GiB 3.5 TiB 61.71 0.98 101 up 159 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 80 KiB 28 GiB 3.5 TiB 61.29 0.98 100 up 163 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 166 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 96 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 145 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 40 KiB 28 GiB 3.5 TiB 61.27 0.98 100 up 148 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 96 KiB 29 GiB 3.4 TiB 62.39 0.99 101 up 151 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB 28 GiB 3.5 TiB 61.70 0.98 101 up 154 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 124 KiB 28 GiB 3.5 TiB 61.23 0.98 100 up 157 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 44 KiB 28 GiB 3.5 TiB 61.32 0.98 100 up 160 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 16 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 162 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 80 KiB 28 GiB 3.5 TiB 61.30 0.98 100 up 165 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 64 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 112 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 28 KiB 29 GiB 3.4 TiB 62.45 0.99 101 up 113 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 28 GiB 3.5 TiB 61.33 0.98 100 up 114 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 107 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 115 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 60 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 116 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 52 KiB 28 GiB 3.5 TiB 61.79 0.98 101 up 117 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 48 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 118 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 60 KiB 29 GiB 3.4 TiB 62.45 0.99 101 up 119 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 96 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 168 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 28 KiB 28 GiB 3.5 TiB 61.29 0.98 100 up 169 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 48 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 170 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 152 KiB 29 GiB 3.4 TiB 62.81 1.00 102 up 171 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 152 KiB 29 GiB 3.4 TiB 62.89 1.00 102 up 172 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 156 KiB 28 GiB 3.5 TiB 61.77 0.98 101 up 173 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 76 KiB 29 GiB 3.4 TiB 62.36 0.99 101 up 174 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 96 KiB 28 GiB 3.5 TiB 61.38 0.98 100 up 175 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 181 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 16 KiB 30 GiB 3.5 TiB 61.30 0.98 100 up 187 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 65 KiB 29 GiB 3.4 TiB 62.41 0.99 101 up 193 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 132 KiB 29 GiB 3.4 TiB 62.40 0.99 101 up 204 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 112 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 210 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 40 KiB 28 GiB 3.5 TiB 61.74 0.98 101 up 220 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 96 KiB 28 GiB 3.5 TiB 61.34 0.98 100 up 222 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 24 KiB 28 GiB 3.5 TiB 61.31 0.98 100 up 223 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 48 KiB 28 GiB 3.5 TiB 61.37 0.98 100 up 92 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.6 TiB 88 KiB 29 GiB 3.4 TiB 62.42 0.99 101 up 93 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 84 KiB 28 GiB 3.5 TiB 61.75 0.98 101 up 94 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 76 KiB 29 GiB 3.4 TiB 62.86 1.00 102 up 95 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 16 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 96 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 8 KiB 29 GiB 3.4 TiB 62.43 0.99 101 up 97 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 72 KiB 29 GiB 3.5 TiB 61.72 0.98 101 up 98 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB 28 GiB 3.5 TiB 61.28 0.98 100 up 99 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB 29 GiB 3.4 TiB 62.85 1.00 102 up 228 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 49 GiB 8.3 GiB 1.3 TiB 63.85 1.02 161 up 229 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 58 GiB 10 GiB 1.3 TiB 63.59 1.01 162 up 230 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 49 GiB 7.4 GiB 1.3 TiB 63.62 1.01 160 up 231 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 50 GiB 7.6 GiB 1.3 TiB 63.71 1.02 161 up 224 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 68 GiB 12 GiB 480 GiB 86.57 1.38 164 up 225 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 46 GiB 11 GiB 513 GiB 85.66 1.36 176 up 226 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.5 TiB 68 GiB 11 GiB 888 GiB 75.18 1.20 166 up 227 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 49 GiB 11 GiB 525 GiB 85.31 1.36 167 up 0 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 58 GiB 10 GiB 1.2 TiB 64.38 1.03 171 up 1 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.1 TiB 64 GiB 11 GiB 1.3 TiB 63.62 1.01 162 up 2 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 66 GiB 11 GiB 1.3 TiB 63.79 1.02 168 up 3 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 64 GiB 11 GiB 1.3 TiB 63.77 1.02 165 up 4 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 64 GiB 12 GiB 876 GiB 75.51 1.20 168 up 6 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 63 GiB 12 GiB 513 GiB 85.65 1.36 160 up 8 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 51 GiB 10 GiB 888 GiB 75.17 1.20 174 up 10 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 63 GiB 13 GiB 498 GiB 86.07 1.37 162 up 5 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 50 GiB 10 GiB 891 GiB 75.07 1.20 164 up 7 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 50 GiB 9.3 GiB 1.3 TiB 63.37 1.01 161 up 9 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.1 TiB 63 GiB 11 GiB 1.3 TiB 63.57 1.01 160 up 11 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 46 GiB 11 GiB 538 GiB 84.94 1.35 168 up 101 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 49 GiB 11 GiB 513 GiB 85.66 1.36 158 up 103 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 50 GiB 10 GiB 900 GiB 74.84 1.19 169 up 105 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.5 TiB 63 GiB 11 GiB 899 GiB 74.85 1.19 163 up 107 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 63 GiB 12 GiB 508 GiB 85.79 1.37 164 up 100 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 50 GiB 11 GiB 534 GiB 85.07 1.36 168 up 102 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.1 TiB 51 GiB 10 GiB 1.3 TiB 63.24 1.01 156 up 104 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 49 GiB 9.5 GiB 1.3 TiB 63.46 1.01 162 up 106 ssd 3.49219 1.00000 3.5 TiB 2.2 TiB 2.2 TiB 65 GiB 11 GiB 1.3 TiB 63.76 1.02 171 up 108 ssd 3.49219 1.00000 3.5 TiB 2.6 TiB 2.6 TiB 66 GiB 12 GiB 874 GiB 75.55 1.20 176 up 109 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 65 GiB 12 GiB 489 GiB 86.32 1.38 166 up 110 ssd 3.49219 1.00000 3.5 TiB 3.4 TiB 3.3 TiB 47 GiB 12 GiB 120 GiB 96.65 1.54 163 up 111 ssd 3.49219 1.00000 3.5 TiB 3.0 TiB 2.9 TiB 66 GiB 12 GiB 518 GiB 85.51 1.36 164 up
OSD 110 is the culprit. I reweighted manually, but its happening agai, all writes only going onthis one OSD. I seen this many times before in luminous and nautilous earlier versions, when will ceph FINALLY fix their pg distribution issue?!
Updated by Neha Ojha about 3 years ago
- Status changed from New to Closed
The balancer is not designed to work during recovery because the final location of the PGs aren't known until recovery finishes. Unfortunately, you will need to wait for PGs to be all active+clean.
Actions