Project

General

Profile

Bug #49197

pg balancer not working during recovery --> bad distribution --> full pools --> complete cluster failure

Added by super xor 20 days ago. Updated 13 days ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

It seems that PG balacner in upmap mode does not work during recovery. Last night we lost a HDD and during the night writes on the SSD pool were heavy and not distributed properly causing complete read-only failure this morning.

 14   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 132 KiB  29 GiB 3.4 TiB 62.18 0.99 102     up 
 19   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 148 KiB  29 GiB 3.4 TiB 62.87 1.00 102     up 
 22   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 108 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 25   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  13 MiB  29 GiB 3.4 TiB 62.46 1.00 102     up 
 30   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  44 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
 33   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  96 KiB  29 GiB 3.4 TiB 62.37 0.99 101     up 
 34   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  56 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
 35   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  52 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
233   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
236   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 160 KiB  29 GiB 3.4 TiB 62.16 0.99 102     up 
239   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  52 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
242   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  68 KiB  28 GiB 3.5 TiB 61.71 0.98 101     up 
244   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  20 KiB  29 GiB 3.4 TiB 62.94 1.00 102     up 
247   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  84 KiB  29 GiB 3.4 TiB 62.37 0.99 101     up 
250   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  24 KiB  29 GiB 3.5 TiB 61.74 0.98 101     up 
253   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 168 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
120   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  24 KiB  29 GiB 3.4 TiB 62.85 1.00 102     up 
122   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  20 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
125   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  28 GiB 3.5 TiB 61.76 0.98 101     up 
127   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  12 KiB  29 GiB 3.5 TiB 61.75 0.98 101     up 
129   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  68 KiB  29 GiB 3.4 TiB 62.86 1.00 102     up 
131   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  84 KiB  28 GiB 3.4 TiB 62.20 0.99 102     up 
133   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
135   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  96 KiB  28 GiB 3.4 TiB 62.19 0.99 102     up 
234   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  29 GiB 3.5 TiB 61.75 0.98 101     up 
238   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB  29 GiB 3.4 TiB 62.88 1.00 102     up 
241   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 116 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
245   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 156 KiB  28 GiB 3.4 TiB 62.15 0.99 102     up 
249   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  60 KiB  29 GiB 3.5 TiB 61.76 0.98 101     up 
252   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 108 KiB  29 GiB 3.5 TiB 61.33 0.98 100     up 
254   hdd 9.09569  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  88 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
255   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  88 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
232   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  52 KiB  29 GiB 3.4 TiB 62.87 1.00 102     up 
235   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  92 KiB  29 GiB 3.4 TiB 62.79 1.00 102     up 
237   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  60 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
240   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
243   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  84 KiB  29 GiB 3.4 TiB 62.50 1.00 101     up 
246   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB   8 KiB  29 GiB 3.4 TiB 62.82 1.00 102     up 
248   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  76 KiB  29 GiB 3.4 TiB 62.81 1.00 102     up 
251   hdd 9.09569  1.00000 9.1 TiB 5.8 TiB 5.7 TiB 156 KiB  29 GiB 3.3 TiB 63.28 1.01 103     up 
 12   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
 16   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  80 KiB  29 GiB 3.4 TiB 62.86 1.00 102     up 
 17   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 116 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
 20   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  92 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
 23   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  16 KiB  28 GiB 3.5 TiB 61.27 0.98 100     up 
 26   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  44 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
 28   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  68 KiB  28 GiB 3.5 TiB 61.35 0.98 100     up 
 31   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  60 KiB  29 GiB 3.4 TiB 62.40 0.99 101     up 
 13   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  28 GiB 3.5 TiB 61.73 0.98 101     up 
 15   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  56 KiB  29 GiB 3.5 TiB 61.33 0.98 100     up 
 18   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  68 KiB  28 GiB 3.5 TiB 61.73 0.98 101     up 
 21   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  76 KiB  28 GiB 3.5 TiB 61.35 0.98 100     up 
 24   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  80 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
 27   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  28 KiB  28 GiB 3.5 TiB 61.60 0.98 100     up 
 29   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
 32   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  88 KiB  29 GiB 3.4 TiB 62.48 1.00 101     up 
 37   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  72 KiB  29 GiB 3.4 TiB 62.82 1.00 102     up 
 39   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.7 TiB  44 KiB  29 GiB 3.3 TiB 63.29 1.01 103     up 
 41   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  72 KiB  29 GiB 3.4 TiB 62.44 0.99 101     up 
 43   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  24 KiB  29 GiB 3.4 TiB 62.79 1.00 102     up 
 45   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  72 KiB  29 GiB 3.4 TiB 62.86 1.00 102     up 
 46   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  64 KiB  29 GiB 3.4 TiB 62.40 0.99 101     up 
 48   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  28 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
 50   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  68 KiB  29 GiB 3.4 TiB 62.39 0.99 101     up 
 36   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 128 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
 38   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  68 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
 40   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  96 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
 42   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  52 KiB  29 GiB 3.4 TiB 62.87 1.00 102     up 
 44   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  68 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
 47   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  28 GiB 3.5 TiB 61.35 0.98 100     up 
 49   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 51   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  20 KiB  28 GiB 3.5 TiB 61.27 0.98 100     up 
 52   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  13 MiB  31 GiB 2.9 TiB 68.22 1.09 106     up 
 53   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 168 KiB  32 GiB 2.9 TiB 68.08 1.08 101     up 
 54   hdd 9.09470        0     0 B     0 B     0 B     0 B     0 B     0 B     0    0   0   down 
 55   hdd 9.09569  1.00000 9.1 TiB 6.1 TiB 6.0 TiB  60 KiB  31 GiB 3.0 TiB 66.82 1.06 105     up 
 56   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.8 TiB 228 KiB  30 GiB 3.2 TiB 64.38 1.03 102     up 
 57   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 332 KiB  31 GiB 3.0 TiB 67.41 1.07 101     up 
 58   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 176 KiB  29 GiB 3.3 TiB 63.54 1.01 102     up 
 59   hdd 9.09569  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 316 KiB  29 GiB 3.2 TiB 64.79 1.03 105     up 
 60   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  76 KiB  30 GiB 3.4 TiB 62.43 0.99 101     up 
 61   hdd 9.09569  1.00000 9.1 TiB 5.8 TiB 5.7 TiB  64 KiB  28 GiB 3.3 TiB 63.25 1.01 103     up 
 62   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  40 KiB  30 GiB 3.4 TiB 62.46 1.00 101     up 
 63   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  72 KiB  29 GiB 3.4 TiB 62.39 0.99 101     up 
 64   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  76 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
 65   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 100 KiB  29 GiB 3.4 TiB 62.36 0.99 101     up 
 66   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  44 KiB  30 GiB 3.4 TiB 62.43 0.99 101     up 
 67   hdd 9.09569  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 216 KiB  31 GiB 3.3 TiB 63.69 1.01 104     up 
 84   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  60 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
 85   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  28 KiB  29 GiB 3.5 TiB 61.33 0.98 100     up 
 86   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  96 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
 87   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  92 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 88   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  28 KiB  29 GiB 3.4 TiB 62.40 0.99 101     up 
 89   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  36 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
 90   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  80 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 91   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  48 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 76   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  68 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
 77   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 100 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
 78   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 136 KiB  28 GiB 3.5 TiB 61.30 0.98 100     up 
 79   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  29 GiB 3.5 TiB 61.38 0.98 100     up 
 80   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  40 KiB  30 GiB 3.4 TiB 62.40 0.99 101     up 
 81   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  92 KiB  29 GiB 3.4 TiB 62.45 0.99 101     up 
 82   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  28 KiB  29 GiB 3.5 TiB 61.37 0.98 100     up 
 83   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
 68   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  80 KiB  29 GiB 3.4 TiB 62.84 1.00 102     up 
 69   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  40 KiB  29 GiB 3.5 TiB 61.27 0.98 100     up 
 70   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  40 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
 71   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  80 KiB  29 GiB 3.4 TiB 62.44 0.99 101     up 
 72   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  28 GiB 3.5 TiB 61.82 0.98 101     up 
 73   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  32 KiB  28 GiB 3.5 TiB 61.27 0.98 100     up 
 74   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 104 KiB  29 GiB 3.4 TiB 62.44 0.99 101     up 
 75   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  56 KiB  29 GiB 3.4 TiB 62.41 0.99 101     up 
180   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  44 KiB  29 GiB 3.4 TiB 62.47 1.00 101     up 
186   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  40 KiB  28 GiB 3.5 TiB 61.76 0.98 101     up 
192   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  28 GiB 3.5 TiB 61.71 0.98 101     up 
197   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  36 KiB  28 GiB 3.5 TiB 61.78 0.98 101     up 
202   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  64 KiB  29 GiB 3.4 TiB 62.38 0.99 101     up 
208   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  80 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
213   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  13 MiB  28 GiB 3.5 TiB 61.74 0.98 102     up 
219   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  96 KiB  28 GiB 3.4 TiB 62.16 0.99 102     up 
176   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  64 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
182   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB  29 GiB 3.4 TiB 62.82 1.00 102     up 
188   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB  28 GiB 3.5 TiB 61.29 0.98 100     up 
196   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  48 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
201   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  28 GiB 3.5 TiB 61.38 0.98 100     up 
207   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
214   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  92 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
218   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  60 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
179   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  76 KiB  28 GiB 3.5 TiB 61.72 0.98 101     up 
185   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 128 KiB  29 GiB 3.4 TiB 62.46 1.00 101     up 
191   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
198   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  44 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
203   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 104 KiB  28 GiB 3.5 TiB 61.75 0.98 101     up 
209   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 120 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
215   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  60 KiB  28 GiB 3.5 TiB 61.81 0.98 101     up 
221   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  88 KiB  28 GiB 3.5 TiB 61.36 0.98 100     up 
177   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  80 KiB  29 GiB 3.4 TiB 62.85 1.00 102     up 
183   hdd 9.09569  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 112 KiB  29 GiB 3.4 TiB 62.81 1.00 102     up 
190   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 148 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
194   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  60 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
199   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 192 KiB  29 GiB 3.4 TiB 62.87 1.00 102     up 
205   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
211   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  28 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
217   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  80 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
178   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  40 KiB  28 GiB 3.5 TiB 61.76 0.98 101     up 
184   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  60 KiB  28 GiB 3.4 TiB 62.35 0.99 101     up 
189   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  56 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
195   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  32 KiB  28 GiB 3.5 TiB 61.18 0.97  99     up 
200   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  80 KiB  29 GiB 3.4 TiB 62.39 0.99 101     up 
206   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 184 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
212   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 116 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
216   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  28 GiB 3.5 TiB 61.79 0.98 101     up 
136   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  92 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
137   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  64 KiB  29 GiB 3.4 TiB 62.47 1.00 101     up 
138   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  52 KiB  29 GiB 3.4 TiB 62.41 0.99 101     up 
139   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  60 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
140   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  36 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
141   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  52 KiB  28 GiB 3.5 TiB 61.30 0.98 100     up 
142   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  52 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
143   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 104 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
121   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  48 KiB  30 GiB 3.4 TiB 62.37 0.99 101     up 
123   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.7 TiB  36 KiB  29 GiB 3.3 TiB 63.34 1.01 103     up 
124   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  44 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
126   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB  29 GiB 3.4 TiB 62.88 1.00 102     up 
128   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 124 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
130   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  80 KiB  29 GiB 3.4 TiB 62.41 0.99 101     up 
132   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  96 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
134   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  61 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
146   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 200 KiB  28 GiB 3.5 TiB 61.75 0.98 101     up 
149   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
152   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  88 KiB  28 GiB 3.4 TiB 62.27 0.99 102     up 
155   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  40 KiB  29 GiB 3.4 TiB 62.44 0.99 101     up 
158   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 108 KiB  28 GiB 3.5 TiB 61.71 0.98 101     up 
161   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  52 KiB  28 GiB 3.5 TiB 61.75 0.98 101     up 
164   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  59 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
167   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  80 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
144   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  76 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
147   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 168 KiB  28 GiB 3.5 TiB 61.35 0.98 100     up 
150   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 108 KiB  29 GiB 3.4 TiB 62.41 0.99 101     up 
153   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  28 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
156   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  92 KiB  28 GiB 3.5 TiB 61.71 0.98 101     up 
159   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  80 KiB  28 GiB 3.5 TiB 61.29 0.98 100     up 
163   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
166   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  96 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
145   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  40 KiB  28 GiB 3.5 TiB 61.27 0.98 100     up 
148   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  96 KiB  29 GiB 3.4 TiB 62.39 0.99 101     up 
151   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB  28 GiB 3.5 TiB 61.70 0.98 101     up 
154   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 124 KiB  28 GiB 3.5 TiB 61.23 0.98 100     up 
157   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  44 KiB  28 GiB 3.5 TiB 61.32 0.98 100     up 
160   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  16 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
162   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  80 KiB  28 GiB 3.5 TiB 61.30 0.98 100     up 
165   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
112   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  28 KiB  29 GiB 3.4 TiB 62.45 0.99 101     up 
113   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  28 GiB 3.5 TiB 61.33 0.98 100     up 
114   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 107 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
115   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  60 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
116   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  52 KiB  28 GiB 3.5 TiB 61.79 0.98 101     up 
117   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  48 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
118   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  60 KiB  29 GiB 3.4 TiB 62.45 0.99 101     up 
119   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  96 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
168   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  28 KiB  28 GiB 3.5 TiB 61.29 0.98 100     up 
169   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  48 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
170   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 152 KiB  29 GiB 3.4 TiB 62.81 1.00 102     up 
171   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 152 KiB  29 GiB 3.4 TiB 62.89 1.00 102     up 
172   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 156 KiB  28 GiB 3.5 TiB 61.77 0.98 101     up 
173   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  76 KiB  29 GiB 3.4 TiB 62.36 0.99 101     up 
174   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  96 KiB  28 GiB 3.5 TiB 61.38 0.98 100     up 
175   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 104 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
181   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  16 KiB  30 GiB 3.5 TiB 61.30 0.98 100     up 
187   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  65 KiB  29 GiB 3.4 TiB 62.41 0.99 101     up 
193   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB 132 KiB  29 GiB 3.4 TiB 62.40 0.99 101     up 
204   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 112 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
210   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  40 KiB  28 GiB 3.5 TiB 61.74 0.98 101     up 
220   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  96 KiB  28 GiB 3.5 TiB 61.34 0.98 100     up 
222   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB  24 KiB  28 GiB 3.5 TiB 61.31 0.98 100     up 
223   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  28 GiB 3.5 TiB 61.37 0.98 100     up 
 92   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  88 KiB  29 GiB 3.4 TiB 62.42 0.99 101     up 
 93   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  84 KiB  28 GiB 3.5 TiB 61.75 0.98 101     up 
 94   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  76 KiB  29 GiB 3.4 TiB 62.86 1.00 102     up 
 95   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  16 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 96   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB   8 KiB  29 GiB 3.4 TiB 62.43 0.99 101     up 
 97   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  29 GiB 3.5 TiB 61.72 0.98 101     up 
 98   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.5 TiB 100 KiB  28 GiB 3.5 TiB 61.28 0.98 100     up 
 99   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB  29 GiB 3.4 TiB 62.85 1.00 102     up 
228   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  49 GiB 8.3 GiB 1.3 TiB 63.85 1.02 161     up 
229   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  58 GiB  10 GiB 1.3 TiB 63.59 1.01 162     up 
230   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  49 GiB 7.4 GiB 1.3 TiB 63.62 1.01 160     up 
231   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  50 GiB 7.6 GiB 1.3 TiB 63.71 1.02 161     up 
224   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  68 GiB  12 GiB 480 GiB 86.57 1.38 164     up 
225   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  46 GiB  11 GiB 513 GiB 85.66 1.36 176     up 
226   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.5 TiB  68 GiB  11 GiB 888 GiB 75.18 1.20 166     up 
227   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  49 GiB  11 GiB 525 GiB 85.31 1.36 167     up 
  0   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  58 GiB  10 GiB 1.2 TiB 64.38 1.03 171     up 
  1   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.1 TiB  64 GiB  11 GiB 1.3 TiB 63.62 1.01 162     up 
  2   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  66 GiB  11 GiB 1.3 TiB 63.79 1.02 168     up 
  3   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  64 GiB  11 GiB 1.3 TiB 63.77 1.02 165     up 
  4   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.6 TiB  64 GiB  12 GiB 876 GiB 75.51 1.20 168     up 
  6   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  63 GiB  12 GiB 513 GiB 85.65 1.36 160     up 
  8   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.6 TiB  51 GiB  10 GiB 888 GiB 75.17 1.20 174     up 
 10   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  63 GiB  13 GiB 498 GiB 86.07 1.37 162     up 
  5   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.6 TiB  50 GiB  10 GiB 891 GiB 75.07 1.20 164     up 
  7   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  50 GiB 9.3 GiB 1.3 TiB 63.37 1.01 161     up 
  9   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.1 TiB  63 GiB  11 GiB 1.3 TiB 63.57 1.01 160     up 
 11   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  46 GiB  11 GiB 538 GiB 84.94 1.35 168     up 
101   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  49 GiB  11 GiB 513 GiB 85.66 1.36 158     up 
103   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.6 TiB  50 GiB  10 GiB 900 GiB 74.84 1.19 169     up 
105   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.5 TiB  63 GiB  11 GiB 899 GiB 74.85 1.19 163     up 
107   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  63 GiB  12 GiB 508 GiB 85.79 1.37 164     up 
100   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  50 GiB  11 GiB 534 GiB 85.07 1.36 168     up 
102   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.1 TiB  51 GiB  10 GiB 1.3 TiB 63.24 1.01 156     up 
104   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  49 GiB 9.5 GiB 1.3 TiB 63.46 1.01 162     up 
106   ssd 3.49219  1.00000 3.5 TiB 2.2 TiB 2.2 TiB  65 GiB  11 GiB 1.3 TiB 63.76 1.02 171     up 
108   ssd 3.49219  1.00000 3.5 TiB 2.6 TiB 2.6 TiB  66 GiB  12 GiB 874 GiB 75.55 1.20 176     up 
109   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  65 GiB  12 GiB 489 GiB 86.32 1.38 166     up 
110   ssd 3.49219  1.00000 3.5 TiB 3.4 TiB 3.3 TiB  47 GiB  12 GiB 120 GiB 96.65 1.54 163     up 
111   ssd 3.49219  1.00000 3.5 TiB 3.0 TiB 2.9 TiB  66 GiB  12 GiB 518 GiB 85.51 1.36 164     up

OSD 110 is the culprit. I reweighted manually, but its happening agai, all writes only going onthis one OSD. I seen this many times before in luminous and nautilous earlier versions, when will ceph FINALLY fix their pg distribution issue?!

History

#1 Updated by Neha Ojha 13 days ago

  • Status changed from New to Closed

The balancer is not designed to work during recovery because the final location of the PGs aren't known until recovery finishes. Unfortunately, you will need to wait for PGs to be all active+clean.

Also available in: Atom PDF