Project

General

Profile

Bug #41313

PG distribution completely messed up since Nautilus

Added by super xor 12 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Backfill/Recovery
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature:

Description

Since using Nautilus the data distribution is an absolute nightmare. We never had issues on Luminous.
Look at this please. I added those empty disks yesterday and nothing happened (rebalance is running but not sure its using them). The worst is OSD 15, it fills up the most, I already had to set it weight to 0 and its still filling it up. I had to down/out it. This should never happen.
And no that PG autobalancer is off because it doens't work whilst cluster is in ERR/WARN state anyway. Not sure what changed in the code but its BAD.

14   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 125 KiB   13 GiB 3.3 TiB 63.95 2.10 112     up 
19   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  88 KiB   12 GiB 3.5 TiB 61.50 2.02 102     up 
22   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 335 KiB   14 GiB 2.9 TiB 68.32 2.24 121     up 
25   hdd 9.09470  1.00000 9.1 TiB 5.1 TiB 5.1 TiB  96 KiB   11 GiB 4.0 TiB 56.07 1.84  97     up 
30   hdd 9.09470  1.00000 9.1 TiB 4.9 TiB 4.9 TiB  64 KiB   11 GiB 4.2 TiB 53.89 1.77  89     up 
33   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  57 KiB   12 GiB 3.5 TiB 62.06 2.03 105     up 
34   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  44 KiB   14 GiB 2.9 TiB 67.93 2.23 121     up 
35   hdd 9.09470  1.00000 9.1 TiB 3.9 TiB 3.9 TiB  19 KiB  9.2 GiB 5.2 TiB 43.24 1.42  71     up 
12   hdd 9.09470  1.00000 9.1 TiB 5.3 TiB 5.3 TiB  56 KiB   12 GiB 3.8 TiB 58.45 1.92 100     up 
16   hdd 9.09470  1.00000 9.1 TiB 4.2 TiB 4.2 TiB  28 KiB  9.8 GiB 4.9 TiB 46.62 1.53  79     up 
17   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB 108 KiB   14 GiB 2.7 TiB 70.81 2.32 119     up 
20   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  48 KiB   13 GiB 3.3 TiB 63.82 2.09 105     up 
23   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  60 KiB   13 GiB 3.4 TiB 62.16 2.04 107     up 
26   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB 195 KiB   14 GiB 2.4 TiB 73.69 2.42 131     up 
28   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  52 KiB   12 GiB 3.4 TiB 62.19 2.04 109     up 
31   hdd 9.09470  1.00000 9.1 TiB 3.8 TiB 3.8 TiB  41 KiB  9.0 GiB 5.3 TiB 41.68 1.37  69     up 
13   hdd 9.09470  1.00000 9.1 TiB 5.4 TiB 5.4 TiB  32 KiB   12 GiB 3.6 TiB 59.91 1.96 102     up 
15   hdd       0  1.00000 9.1 TiB 7.6 TiB 7.6 TiB 112 KiB   16 GiB 1.5 TiB 83.69 2.74 147     up 
18   hdd 9.09470  1.00000 9.1 TiB 4.7 TiB 4.7 TiB 198 KiB   10 GiB 4.4 TiB 51.60 1.69  87     up 
21   hdd 9.09470  1.00000 9.1 TiB 4.3 TiB 4.3 TiB   4 KiB   10 GiB 4.8 TiB 47.49 1.56  77     up 
24   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB   14 GiB 2.8 TiB 68.91 2.26 122     up 
27   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB 298 KiB   13 GiB 3.1 TiB 65.78 2.16 113     up 
29   hdd 9.09470  1.00000 9.1 TiB 5.2 TiB 5.2 TiB 147 KiB   11 GiB 3.9 TiB 57.38 1.88 102     up 
32   hdd 9.09470  1.00000 9.1 TiB 4.6 TiB 4.6 TiB 317 KiB   10 GiB 4.5 TiB 50.64 1.66  84     up 
37   hdd 9.09470  1.00000 9.1 TiB 2.1 TiB 2.1 TiB  32 KiB  5.0 GiB 7.0 TiB 22.62 0.74  31     up 
39   hdd 9.09470  1.00000 9.1 TiB 1.8 TiB 1.8 TiB  88 KiB  4.2 GiB 7.3 TiB 19.32 0.63  29     up 
41   hdd 9.09470  1.00000 9.1 TiB 1.9 TiB 1.9 TiB  44 KiB  4.5 GiB 7.2 TiB 21.13 0.69  32     up 
43   hdd 9.09470  1.00000 9.1 TiB 2.8 TiB 2.8 TiB  56 KiB  6.3 GiB 6.3 TiB 30.67 1.01  43     up 
45   hdd 9.09470  1.00000 9.1 TiB 2.9 TiB 2.9 TiB  48 KiB  6.8 GiB 6.2 TiB 32.09 1.05  53     up 
46   hdd 9.09470  1.00000 9.1 TiB 2.7 TiB 2.7 TiB  48 KiB  6.3 GiB 6.4 TiB 29.54 0.97  45     up 
48   hdd 9.09470  1.00000 9.1 TiB 2.5 TiB 2.5 TiB  44 KiB  5.7 GiB 6.6 TiB 27.77 0.91  41     up 
50   hdd 9.09470  1.00000 9.1 TiB 1.9 TiB 1.9 TiB  76 KiB  4.4 GiB 7.2 TiB 21.01 0.69  28     up 
36   hdd 9.09470  1.00000 9.1 TiB 1.7 TiB 1.7 TiB  32 KiB  4.1 GiB 7.4 TiB 18.77 0.62  31     up 
38   hdd 9.09470  1.00000 9.1 TiB 2.3 TiB 2.3 TiB  28 KiB  5.5 GiB 6.8 TiB 25.44 0.83  37     up 
40   hdd 9.09470  1.00000 9.1 TiB 2.2 TiB 2.2 TiB  56 KiB  5.1 GiB 6.9 TiB 24.52 0.80  40     up 
42   hdd 9.09470  1.00000 9.1 TiB 2.3 TiB 2.2 TiB  60 KiB  5.2 GiB 6.8 TiB 24.78 0.81  34     up 
44   hdd 9.09470  1.00000 9.1 TiB 2.6 TiB 2.6 TiB  36 KiB  5.9 GiB 6.5 TiB 28.75 0.94  46     up 
47   hdd 9.09470  1.00000 9.1 TiB 2.6 TiB 2.6 TiB  64 KiB  5.9 GiB 6.5 TiB 28.69 0.94  42     up 
49   hdd 9.09470  1.00000 9.1 TiB 2.1 TiB 2.1 TiB  52 KiB  4.8 GiB 7.0 TiB 22.79 0.75  34     up 
51   hdd 9.09470  1.00000 9.1 TiB 2.1 TiB 2.1 TiB  36 KiB  4.9 GiB 7.0 TiB 23.44 0.77  35     up 
52   hdd 9.09470  1.00000 9.1 TiB  15 GiB  14 GiB  28 KiB 1024 MiB 9.1 TiB  0.17 0.01   0     up 
53   hdd 9.09470  1.00000 9.1 TiB  20 GiB  19 GiB   4 KiB 1024 MiB 9.1 TiB  0.22 0.01   0     up 
54   hdd 9.09470  1.00000 9.1 TiB  11 GiB  10 GiB  28 KiB 1024 MiB 9.1 TiB  0.12 0.00   0     up 
55   hdd 9.09470  1.00000 9.1 TiB  17 GiB  16 GiB   4 KiB 1024 MiB 9.1 TiB  0.18 0.01   0     up 
56   hdd 9.09470  1.00000 9.1 TiB  19 GiB  18 GiB   8 KiB 1024 MiB 9.1 TiB  0.20 0.01   0     up 
57   hdd 9.09470  1.00000 9.1 TiB  21 GiB  20 GiB   4 KiB 1024 MiB 9.1 TiB  0.22 0.01   0     up 
58   hdd 9.09470  1.00000 9.1 TiB  21 GiB  20 GiB  12 KiB 1024 MiB 9.1 TiB  0.22 0.01   0     up 
59   hdd 9.09470  1.00000 9.1 TiB  21 GiB  20 GiB  20 KiB 1024 MiB 9.1 TiB  0.22 0.01   0     up 
60   hdd 9.09470  1.00000 9.1 TiB  19 GiB  18 GiB   4 KiB 1024 MiB 9.1 TiB  0.21 0.01   0     up 
61   hdd 9.09470  1.00000 9.1 TiB  13 GiB  12 GiB  20 KiB 1024 MiB 9.1 TiB  0.14 0.00   0     up 
62   hdd 9.09470  1.00000 9.1 TiB  19 GiB  18 GiB  28 KiB 1024 MiB 9.1 TiB  0.20 0.01   0     up 
63   hdd 9.09470  1.00000 9.1 TiB  20 GiB  19 GiB  16 KiB 1024 MiB 9.1 TiB  0.21 0.01   0     up 
64   hdd 9.09470  1.00000 9.1 TiB  19 GiB  18 GiB  28 KiB 1024 MiB 9.1 TiB  0.20 0.01   0     up 
65   hdd 9.09470  1.00000 9.1 TiB  19 GiB  18 GiB  20 KiB 1024 MiB 9.1 TiB  0.21 0.01   0     up 
66   hdd 9.09470  1.00000 9.1 TiB  20 GiB  19 GiB   8 KiB 1024 MiB 9.1 TiB  0.21 0.01   0     up 
67   hdd 9.09470  1.00000 9.1 TiB  20 GiB  19 GiB  12 KiB 1024 MiB 9.1 TiB  0.22 0.01   0     up
ID  CLASS WEIGHT    TYPE NAME                     STATUS REWEIGHT PRI-AFF 
 -1       542.11450 root default                                          
-19        72.75757     host fsn-1-dc4-s3-1035753                         
 14   hdd   9.09470         osd.14                    up  1.00000 1.00000 
 19   hdd   9.09470         osd.19                    up  1.00000 1.00000 
 22   hdd   9.09470         osd.22                    up  1.00000 1.00000 
 25   hdd   9.09470         osd.25                    up  1.00000 1.00000 
 30   hdd   9.09470         osd.30                    up  1.00000 1.00000 
 33   hdd   9.09470         osd.33                    up  1.00000 1.00000 
 34   hdd   9.09470         osd.34                    up  1.00000 1.00000 
 35   hdd   9.09470         osd.35                    up  1.00000 1.00000 
-16        72.75757     host fsn-1-dc4-s3-1060345                         
 12   hdd   9.09470         osd.12                    up  1.00000 1.00000 
 16   hdd   9.09470         osd.16                    up  1.00000 1.00000 
 17   hdd   9.09470         osd.17                    up  1.00000 1.00000 
 20   hdd   9.09470         osd.20                    up  1.00000 1.00000 
 23   hdd   9.09470         osd.23                    up  1.00000 1.00000 
 26   hdd   9.09470         osd.26                    up  1.00000 1.00000 
 28   hdd   9.09470         osd.28                    up  1.00000 1.00000 
 31   hdd   9.09470         osd.31                    up  1.00000 1.00000 
-13        63.66287     host fsn-1-dc4-s3-1060346                         
 13   hdd   9.09470         osd.13                    up  1.00000 1.00000 
 15   hdd         0         osd.15                    up  1.00000 1.00000 
 18   hdd   9.09470         osd.18                    up  1.00000 1.00000 
 21   hdd   9.09470         osd.21                    up  1.00000 1.00000 
 24   hdd   9.09470         osd.24                    up  1.00000 1.00000 
 27   hdd   9.09470         osd.27                    up  1.00000 1.00000 
 29   hdd   9.09470         osd.29                    up  1.00000 1.00000 
 32   hdd   9.09470         osd.32                    up  1.00000 1.00000 
-22        72.75757     host fsn-1-dc4-s3-1065722                         
 37   hdd   9.09470         osd.37                    up  1.00000 1.00000 
 39   hdd   9.09470         osd.39                    up  1.00000 1.00000 
 41   hdd   9.09470         osd.41                    up  1.00000 1.00000 
 43   hdd   9.09470         osd.43                    up  1.00000 1.00000 
 45   hdd   9.09470         osd.45                    up  1.00000 1.00000 
 46   hdd   9.09470         osd.46                    up  1.00000 1.00000 
 48   hdd   9.09470         osd.48                    up  1.00000 1.00000 
 50   hdd   9.09470         osd.50                    up  1.00000 1.00000 
-25        72.75757     host fsn-1-dc4-s3-1065723                         
 36   hdd   9.09470         osd.36                    up  1.00000 1.00000 
 38   hdd   9.09470         osd.38                    up  1.00000 1.00000 
 40   hdd   9.09470         osd.40                    up  1.00000 1.00000 
 42   hdd   9.09470         osd.42                    up  1.00000 1.00000 
 44   hdd   9.09470         osd.44                    up  1.00000 1.00000 
 47   hdd   9.09470         osd.47                    up  1.00000 1.00000 
 49   hdd   9.09470         osd.49                    up  1.00000 1.00000 
 51   hdd   9.09470         osd.51                    up  1.00000 1.00000 
-28        72.75757     host fsn-1-dc4-s3-1066329                         
 52   hdd   9.09470         osd.52                    up  1.00000 1.00000 
 53   hdd   9.09470         osd.53                    up  1.00000 1.00000 
 54   hdd   9.09470         osd.54                    up  1.00000 1.00000 
 55   hdd   9.09470         osd.55                    up  1.00000 1.00000 
 56   hdd   9.09470         osd.56                    up  1.00000 1.00000 
 57   hdd   9.09470         osd.57                    up  1.00000 1.00000 
 58   hdd   9.09470         osd.58                    up  1.00000 1.00000 
 59   hdd   9.09470         osd.59                    up  1.00000 1.00000 
-31        72.75757     host fsn-1-dc4-s3-1066330                         
 60   hdd   9.09470         osd.60                    up  1.00000 1.00000 
 61   hdd   9.09470         osd.61                    up  1.00000 1.00000 
 62   hdd   9.09470         osd.62                    up  1.00000 1.00000 
 63   hdd   9.09470         osd.63                    up  1.00000 1.00000 
 64   hdd   9.09470         osd.64                    up  1.00000 1.00000 
 65   hdd   9.09470         osd.65                    up  1.00000 1.00000 
 66   hdd   9.09470         osd.66                    up  1.00000 1.00000 
 67   hdd   9.09470         osd.67                    up  1.00000 1.00000
    pgs:     6515347/169178931 objects degraded (3.851%)
             77130467/169178931 objects misplaced (45.591%)
             652 active+remapped+backfill_wait
             322 active+clean
             89  active+undersized+degraded+remapped+backfill_wait
             58  active+undersized+degraded+remapped+backfilling
             54  active+remapped+backfill_wait+backfill_toofull
             33  active+remapped+backfilling

History

#1 Updated by Neha Ojha 12 months ago

Can you reach out on the ceph-users mailing list to see if others have seen similar issues? We've not seen a specific bug related to this.

#2 Updated by super xor 9 months ago

There definitely is something wrong.
After every kind of rebalance, no matter if caused by OSD removal/adding, or now PG raise, the PG distribution gets worse and worse. I was raising pool PGs every day bit by bit. First it was 1 nearfull OSD, then 4, now it's 8!.

 14   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  16 GiB 3.5 TiB 61.34 0.86  35     up 
 19   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.8 TiB  12 KiB  18 GiB 2.2 TiB 75.43 1.06  39     up 
 22   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  20 KiB  17 GiB 2.8 TiB 69.57 0.98  38     up 
 25   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 8.0 TiB 1.0 MiB  21 GiB 1.1 TiB 88.03 1.24  50     up 
 30   hdd 9.09470  1.00000 9.1 TiB 7.5 TiB 7.5 TiB  96 KiB  19 GiB 1.6 TiB 82.62 1.16  48     up 
 33   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  64 KiB  18 GiB 2.2 TiB 75.97 1.07  36     up 
 34   hdd 9.09470  1.00000 9.1 TiB 7.1 TiB 7.1 TiB  60 KiB  18 GiB 2.0 TiB 78.37 1.10  39     up 
 35   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB  16 KiB  19 GiB 1.8 TiB 80.49 1.13  42     up 
 12   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  32 KiB  18 GiB 2.5 TiB 72.01 1.01  35     up 
 16   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  60 KiB  19 GiB 2.1 TiB 77.27 1.09  42     up 
 17   hdd 9.09470  1.00000 9.1 TiB 7.5 TiB 7.5 TiB  88 KiB  20 GiB 1.6 TiB 82.68 1.17  49     up 
 20   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  72 KiB  17 GiB 2.7 TiB 70.52 0.99  39     up 
 23   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  60 KiB  18 GiB 2.1 TiB 77.39 1.09  47     up 
 26   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  36 KiB  16 GiB 3.1 TiB 66.46 0.94  35     up 
 28   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.7 TiB  32 KiB  16 GiB 3.3 TiB 63.34 0.89  35     up 
 31   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  12 KiB  18 GiB 2.6 TiB 71.67 1.01  35     up 
 13   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  56 KiB  17 GiB 2.4 TiB 73.30 1.03  37     up 
 15   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  18 GiB 2.4 TiB 73.34 1.03  43     up 
 18   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB 108 KiB  18 GiB 2.1 TiB 77.32 1.09  38     up 
 21   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 8.0 TiB  64 KiB  22 GiB 1.1 TiB 88.38 1.25  44     up 
 24   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  84 KiB  16 GiB 2.9 TiB 67.62 0.95  35     up 
 27   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  40 KiB  18 GiB 2.2 TiB 76.03 1.07  43     up 
 29   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  60 KiB  16 GiB 3.3 TiB 64.02 0.90  38     up 
 32   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 8.0 TiB  52 KiB  21 GiB 1.1 TiB 88.20 1.24  50     up 
 37   hdd 9.09470  1.00000 9.1 TiB 7.7 TiB 7.7 TiB 100 KiB  20 GiB 1.4 TiB 84.89 1.20  48     up 
 39   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  20 KiB  18 GiB 2.4 TiB 73.34 1.03  38     up 
 41   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  84 KiB  18 GiB 2.4 TiB 73.47 1.04  36     up 
 43   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  60 KiB  17 GiB 2.9 TiB 68.28 0.96  37     up 
 45   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB  17 GiB 2.8 TiB 69.57 0.98  40     up 
 46   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  72 KiB  17 GiB 2.7 TiB 70.43 0.99  36     up 
 48   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  36 KiB  19 GiB 2.3 TiB 74.64 1.05  38     up 
 50   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  84 KiB  17 GiB 2.6 TiB 71.48 1.01  37     up 
 36   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB 100 KiB  17 GiB 2.8 TiB 69.76 0.98  38     up 
 38   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  17 GiB 2.9 TiB 68.03 0.96  37     up 
 40   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  52 KiB  19 GiB 2.4 TiB 73.30 1.03  41     up 
 42   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  44 KiB  18 GiB 2.3 TiB 74.48 1.05  40     up 
 44   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  84 KiB  18 GiB 2.2 TiB 76.27 1.07  36     up 
 47   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  36 KiB  19 GiB 2.6 TiB 71.56 1.01  39     up 
 49   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  72 KiB  15 GiB 3.5 TiB 61.36 0.86  32     up 
 51   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.8 TiB  28 KiB  18 GiB 2.2 TiB 75.40 1.06  38     up 
 52   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB 1.1 MiB  17 GiB 2.7 TiB 70.46 0.99  39     up 
 53   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  28 KiB  19 GiB 1.9 TiB 79.23 1.12  30     up 
 54   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  88 KiB  16 GiB 2.9 TiB 67.83 0.96  33     up 
 55   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  60 KiB  18 GiB 2.2 TiB 76.06 1.07  46     up 
 56   hdd 9.09470  1.00000 9.1 TiB 7.4 TiB 7.4 TiB  93 KiB  19 GiB 1.7 TiB 81.54 1.15  40     up 
 57   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  60 KiB  18 GiB 2.2 TiB 76.14 1.07  38     up 
 58   hdd 9.09470  1.00000 9.1 TiB 7.4 TiB 7.4 TiB  76 KiB  19 GiB 1.7 TiB 81.02 1.14  44     up 
 59   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  84 KiB  19 GiB 2.3 TiB 74.95 1.06  42     up 
 60   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  60 KiB  17 GiB 2.7 TiB 70.40 0.99  39     up 
 61   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB  16 KiB  19 GiB 1.8 TiB 80.09 1.13  45     up 
 62   hdd 9.09470  1.00000 9.1 TiB 7.1 TiB 7.1 TiB  96 KiB  18 GiB 2.0 TiB 78.39 1.10  46     up 
 63   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB 136 KiB  18 GiB 2.5 TiB 73.03 1.03  41     up 
 64   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  36 KiB  18 GiB 2.1 TiB 77.16 1.09  39     up 
 65   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB 144 KiB  18 GiB 2.2 TiB 75.83 1.07  41     up 
 66   hdd 9.09470  1.00000 9.1 TiB 7.7 TiB 7.7 TiB 104 KiB  20 GiB 1.4 TiB 84.43 1.19  41     up 
 67   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB 108 KiB  19 GiB 1.8 TiB 80.46 1.13  43     up 
 84   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  76 KiB  18 GiB 2.2 TiB 75.61 1.07  42     up 
 85   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  72 KiB  19 GiB 1.9 TiB 79.25 1.12  43     up 
 86   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB  32 KiB  19 GiB 1.8 TiB 80.40 1.13  45     up 
 87   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  48 KiB  18 GiB 2.5 TiB 72.34 1.02  40     up 
 88   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB  32 KiB  19 GiB 1.8 TiB 79.95 1.13  45     up 
 89   hdd 9.09470  1.00000 9.1 TiB 7.4 TiB 7.3 TiB  40 KiB  20 GiB 1.7 TiB 81.02 1.14  44     up 
 90   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  28 KiB  17 GiB 3.0 TiB 66.48 0.94  36     up 
 91   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  96 KiB  16 GiB 2.9 TiB 67.92 0.96  37     up 
 76   hdd 9.09470  1.00000 9.1 TiB 7.4 TiB 7.4 TiB  48 KiB  19 GiB 1.7 TiB 81.26 1.14  42     up 
 77   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  48 KiB  17 GiB 3.1 TiB 66.28 0.93  37     up 
 78   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  89 KiB  17 GiB 2.8 TiB 69.66 0.98  39     up 
 79   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  32 KiB  17 GiB 2.8 TiB 69.22 0.98  40     up 
 80   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  76 KiB  18 GiB 2.5 TiB 72.68 1.02  36     up 
 81   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 112 KiB  16 GiB 2.9 TiB 67.93 0.96  35     up 
 82   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  18 GiB 2.3 TiB 74.52 1.05  38     up 
 83   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 7.9 TiB  84 KiB  21 GiB 1.1 TiB 87.43 1.23  45     up 
 68   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB  24 KiB  18 GiB 2.6 TiB 71.01 1.00  36     up 
 69   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  20 KiB  20 GiB 1.9 TiB 79.06 1.11  38     up 
 70   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  60 KiB  18 GiB 2.4 TiB 73.13 1.03  39     up 
 71   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 6.9 TiB  32 KiB  19 GiB 2.1 TiB 76.58 1.08  43     up 
 72   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.5 TiB  36 KiB  17 GiB 2.5 TiB 72.04 1.02  39     up 
 73   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  56 KiB  16 GiB 3.0 TiB 67.55 0.95  32     up 
 74   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB   8 KiB  18 GiB 2.4 TiB 73.25 1.03  42     up 
 75   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  52 KiB  17 GiB 3.1 TiB 65.90 0.93  33     up 
121   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  72 KiB  18 GiB 2.7 TiB 70.71 1.00  39     up 
123   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 8.0 TiB  88 KiB  20 GiB 1.1 TiB 87.65 1.23  49     up 
124   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  76 KiB  17 GiB 2.9 TiB 68.20 0.96  35     up 
126   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  56 KiB  18 GiB 2.0 TiB 77.51 1.09  46     up 
128   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  28 KiB  18 GiB 2.2 TiB 75.69 1.07  40     up 
130   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB 104 KiB  18 GiB 1.8 TiB 79.71 1.12  37     up 
132   hdd 9.09470  1.00000 9.1 TiB 7.5 TiB 7.5 TiB  28 KiB  19 GiB 1.6 TiB 82.35 1.16  42     up 
134   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  16 KiB  17 GiB 3.2 TiB 65.11 0.92  36     up 
112   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  52 KiB  19 GiB 2.2 TiB 75.60 1.07  40     up 
113   hdd 9.09470  1.00000 9.1 TiB 7.4 TiB 7.3 TiB  88 KiB  19 GiB 1.7 TiB 81.00 1.14  43     up 
114   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  36 KiB  18 GiB 2.4 TiB 73.71 1.04  43     up 
115   hdd 9.09470  1.00000 9.1 TiB 7.6 TiB 7.6 TiB  32 KiB  20 GiB 1.5 TiB 83.99 1.18  43     up 
116   hdd 9.09470  1.00000 9.1 TiB 7.5 TiB 7.4 TiB  92 KiB  19 GiB 1.6 TiB 81.97 1.15  42     up 
117   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.1 TiB  68 KiB  19 GiB 1.9 TiB 78.63 1.11  44     up 
118   hdd 9.09470  1.00000 9.1 TiB 7.9 TiB 7.9 TiB 128 KiB  20 GiB 1.2 TiB 87.25 1.23  40     up 
119   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  28 KiB  18 GiB 2.1 TiB 76.39 1.08  37     up 
120   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  52 KiB  17 GiB 2.6 TiB 71.55 1.01  39     up 
122   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  28 KiB  18 GiB 2.1 TiB 76.36 1.08  37     up 
125   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  64 KiB  19 GiB 2.0 TiB 77.49 1.09  42     up 
127   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.5 TiB 1.1 MiB  18 GiB 2.5 TiB 72.14 1.02  44     up 
129   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  64 KiB  19 GiB 1.9 TiB 78.86 1.11  45     up 
131   hdd 9.09470  1.00000 9.1 TiB 7.1 TiB 7.1 TiB  44 KiB  19 GiB 2.0 TiB 78.55 1.11  43     up 
133   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  36 KiB  18 GiB 2.5 TiB 73.01 1.03  38     up 
135   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB 112 KiB  18 GiB 2.2 TiB 76.06 1.07  39     up 
 92   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  17 GiB 2.4 TiB 73.87 1.04  41     up 
 93   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  32 KiB  18 GiB 2.4 TiB 73.95 1.04  43     up 
 94   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  80 KiB  16 GiB 3.1 TiB 66.46 0.94  35     up 
 95   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  72 KiB  16 GiB 2.9 TiB 67.58 0.95  35     up 
 96   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  92 KiB  17 GiB 2.9 TiB 68.22 0.96  37     up 
 97   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  92 KiB  18 GiB 2.1 TiB 77.41 1.09  39     up 
 98   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  36 KiB  19 GiB 1.9 TiB 79.51 1.12  37     up 
 99   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  32 KiB  17 GiB 2.4 TiB 73.31 1.03  41     up 

Which means if someone does a bigger rebalance or PG raise there's a risk of OSDs running full causing pool shutdown.
This NEVER happend under Luminous.

#3 Updated by super xor 8 months ago

This happens with active PG balancer if the cluster is in WARN state.

...
51 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 100 KiB 15 GiB 3.3 TiB 64.01 0.98 24 up
52 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 3.1 MiB 16 GiB 2.8 TiB 69.62 1.07 28 up
53 hdd 9.09470 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 32 KiB 21 GiB 1.0 TiB 88.47 1.36 31 up
54 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 100 KiB 17 GiB 2.4 TiB 73.44 1.13 28 up
55 hdd 9.09470 1.00000 9.1 TiB 5.5 TiB 5.5 TiB 40 KiB 15 GiB 3.6 TiB 60.27 0.93 27 up
...

Major mismatch here.

I posted about this so many times now, I replicated it, nobody seems to bother. It's a MAJOR issue.

#4 Updated by super xor 8 months ago

ceph balancer status {
"active": true,
"plans": [],
"mode": "upmap"
}

bad distribution:

  0   ssd 3.49219  1.00000 3.5 TiB 713 GiB 673 GiB  36 GiB 3.6 GiB 2.8 TiB 19.93 0.29 147     up 
  1   ssd 3.49219  1.00000 3.5 TiB 724 GiB 671 GiB  49 GiB 3.9 GiB 2.8 TiB 20.24 0.30 146     up 
  2   ssd 3.49219  1.00000 3.5 TiB 735 GiB 680 GiB  50 GiB 4.2 GiB 2.8 TiB 20.54 0.30 150     up 
  3   ssd 3.49219  1.00000 3.5 TiB 712 GiB 675 GiB  33 GiB 3.5 GiB 2.8 TiB 19.91 0.29 146     up 
  4   ssd 3.49219  1.00000 3.5 TiB 752 GiB 713 GiB  34 GiB 4.8 GiB 2.8 TiB 21.02 0.31 156     up 
  6   ssd 3.49219  1.00000 3.5 TiB 709 GiB 670 GiB  35 GiB 3.8 GiB 2.8 TiB 19.83 0.29 146     up 
  8   ssd 3.49219  1.00000 3.5 TiB 781 GiB 737 GiB  40 GiB 3.7 GiB 2.7 TiB 21.83 0.32 156     up 
 10   ssd 3.49219  1.00000 3.5 TiB 727 GiB 681 GiB  42 GiB 3.8 GiB 2.8 TiB 20.33 0.30 146     up 
  5   ssd 3.49219  1.00000 3.5 TiB 664 GiB 628 GiB  32 GiB 4.3 GiB 2.8 TiB 18.57 0.27 141     up 
  7   ssd 3.49219  1.00000 3.5 TiB 656 GiB 613 GiB  39 GiB 4.1 GiB 2.9 TiB 18.35 0.27 136     up 
  9   ssd 3.49219  1.00000 3.5 TiB 631 GiB 585 GiB  41 GiB 4.0 GiB 2.9 TiB 17.64 0.26 131     up 
 11   ssd 3.49219  1.00000 3.5 TiB 725 GiB 701 GiB  22 GiB 2.4 GiB 2.8 TiB 20.26 0.30 138     up 
101   ssd 3.49219  1.00000 3.5 TiB 754 GiB 712 GiB  38 GiB 4.0 GiB 2.8 TiB 21.09 0.31 146     up 
103   ssd 3.49219  1.00000 3.5 TiB 761 GiB 717 GiB  40 GiB 3.5 GiB 2.7 TiB 21.28 0.31 150     up 
105   ssd 3.49219  1.00000 3.5 TiB 715 GiB 676 GiB  36 GiB 2.8 GiB 2.8 TiB 19.98 0.29 148     up 
107   ssd 3.49219  1.00000 3.5 TiB 759 GiB 706 GiB  50 GiB 3.2 GiB 2.8 TiB 21.23 0.31 147     up 
100   ssd 3.49219  1.00000 3.5 TiB 724 GiB 674 GiB  47 GiB 2.4 GiB 2.8 TiB 20.24 0.30 144     up 
102   ssd 3.49219  1.00000 3.5 TiB 669 GiB 654 GiB  12 GiB 2.5 GiB 2.8 TiB 18.70 0.27 141     up 
104   ssd 3.49219  1.00000 3.5 TiB 720 GiB 687 GiB  31 GiB 3.0 GiB 2.8 TiB 20.14 0.30 144     up 
106   ssd 3.49219  1.00000 3.5 TiB 713 GiB 645 GiB  65 GiB 3.2 GiB 2.8 TiB 19.95 0.29 143     up 
108   ssd 3.49219  1.00000 3.5 TiB 728 GiB 690 GiB  36 GiB 2.6 GiB 2.8 TiB 20.36 0.30 156     up 
109   ssd 3.49219  1.00000 3.5 TiB 731 GiB 684 GiB  45 GiB 3.0 GiB 2.8 TiB 20.46 0.30 146     up 
110   ssd 3.49219  1.00000 3.5 TiB 773 GiB 742 GiB  28 GiB 2.6 GiB 2.7 TiB 21.62 0.32 154     up 
111   ssd 3.49219  1.00000 3.5 TiB 707 GiB 659 GiB  45 GiB 2.7 GiB 2.8 TiB 19.76 0.29 146     up

#5 Updated by super xor 7 months ago

After being told on the ML that my low PG count is supposed to be the reason, I raised it. Nothing changed, still same issues here.

 14   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB 8.0 TiB  96 KiB  21 GiB 1.1 TiB 88.17 1.34 159     up 
 19   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  72 KiB  17 GiB 2.8 TiB 69.29 1.05 126     up 
 22   hdd 9.09470  1.00000 9.1 TiB 7.9 TiB 7.9 TiB 132 KiB  21 GiB 1.2 TiB 87.15 1.33 156     up 
 25   hdd 9.09470  1.00000 9.1 TiB 7.3 TiB 7.3 TiB 6.4 MiB  19 GiB 1.8 TiB 80.14 1.22 140     up 
 30   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  88 KiB  17 GiB 2.4 TiB 73.39 1.12 130     up 
 33   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  36 KiB  18 GiB 2.3 TiB 74.47 1.13 131     up 
 34   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  96 KiB  19 GiB 1.9 TiB 79.18 1.20 142     up 
 35   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  17 GiB 2.7 TiB 70.64 1.07 129     up 
 12   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  56 KiB  17 GiB 2.5 TiB 72.25 1.10 129     up 
 16   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  76 KiB  18 GiB 2.2 TiB 76.22 1.16 137     up 
 17   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB 112 KiB  18 GiB 2.1 TiB 77.11 1.17 142     up 
 20   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  72 KiB  17 GiB 2.8 TiB 69.42 1.06 125     up 
 23   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  56 KiB  18 GiB 2.5 TiB 72.35 1.10 131     up 
 26   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  17 GiB 2.8 TiB 69.50 1.06 126     up 
 28   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 116 KiB  17 GiB 3.2 TiB 64.59 0.98 119     up 
 31   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.7 TiB  56 KiB  16 GiB 3.3 TiB 63.28 0.96 114     up 
 13   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  44 KiB  18 GiB 2.8 TiB 69.23 1.05 126     up 
 15   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  48 KiB  16 GiB 3.1 TiB 66.27 1.01 121     up 
 18   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB   8 KiB  17 GiB 2.8 TiB 69.64 1.06 124     up 
 21   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB 132 KiB  18 GiB 2.1 TiB 76.81 1.17 142     up 
 24   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  16 GiB 3.2 TiB 65.08 0.99 118     up 
 27   hdd 9.09470  1.00000 9.1 TiB 7.2 TiB 7.2 TiB  12 KiB  19 GiB 1.9 TiB 78.82 1.20 144     up 
 29   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 108 KiB  16 GiB 3.2 TiB 65.22 0.99 117     up 
 32   hdd 9.09470  1.00000 9.1 TiB 7.5 TiB 7.5 TiB 124 KiB  19 GiB 1.6 TiB 82.74 1.26 150     up 
 37   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  56 KiB  16 GiB 3.2 TiB 65.34 0.99 118     up 
 39   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  44 KiB  18 GiB 2.2 TiB 76.05 1.16 134     up 
 41   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB  16 GiB 3.4 TiB 62.90 0.96 113     up 
 43   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  32 KiB  18 GiB 2.4 TiB 73.15 1.11 129     up 
 45   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 116 KiB  16 GiB 3.4 TiB 62.72 0.95 113     up 
 46   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 136 KiB  17 GiB 2.9 TiB 68.46 1.04 124     up 
 48   hdd 9.09470  1.00000 9.1 TiB 5.2 TiB 5.2 TiB  56 KiB  15 GiB 3.9 TiB 57.64 0.88 104     up 
 50   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  16 GiB 3.0 TiB 67.37 1.02 120     up 
 36   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  18 GiB 2.3 TiB 74.60 1.13 134     up 
 38   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB 120 KiB  18 GiB 2.4 TiB 73.83 1.12 131     up 
 40   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  48 KiB  18 GiB 2.5 TiB 72.69 1.11 131     up 
 42   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  48 KiB  18 GiB 2.5 TiB 72.01 1.10 129     up 
 44   hdd 9.09470  1.00000 9.1 TiB 5.5 TiB 5.4 TiB 104 KiB  15 GiB 3.6 TiB 60.06 0.91 109     up 
 47   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB  48 KiB  17 GiB 2.6 TiB 70.99 1.08 130     up 
 49   hdd 9.09470  1.00000 9.1 TiB 5.2 TiB 5.2 TiB  92 KiB  15 GiB 3.9 TiB 57.54 0.88 102     up 
 51   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  72 KiB  16 GiB 3.2 TiB 64.81 0.99 116     up 
 52   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB 6.3 MiB  18 GiB 2.3 TiB 74.18 1.13 134     up 
 53   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 104 KiB  16 GiB 3.2 TiB 64.99 0.99 114     up 
 54   hdd 9.09470  1.00000 9.1 TiB 5.4 TiB 5.4 TiB  52 KiB  15 GiB 3.7 TiB 59.57 0.91 108     up 
 55   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 6.9 TiB  52 KiB  19 GiB 2.1 TiB 76.56 1.16 136     up 
 56   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  72 KiB  18 GiB 2.6 TiB 71.86 1.09 125     up 
 57   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB  16 GiB 3.3 TiB 63.59 0.97 115     up 
 58   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB  15 GiB 3.5 TiB 61.75 0.94 110     up 
 59   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  60 KiB  18 GiB 2.4 TiB 73.20 1.11 132     up 
 60   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  60 KiB  16 GiB 3.2 TiB 65.00 0.99 116     up 
 61   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB 112 KiB  16 GiB 3.1 TiB 65.86 1.00 117     up 
 62   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB 120 KiB  18 GiB 2.6 TiB 71.59 1.09 127     up 
 63   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  16 KiB  17 GiB 2.9 TiB 68.06 1.04 120     up 
 64   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.7 TiB  96 KiB  18 GiB 2.3 TiB 74.30 1.13 133     up 
 65   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  18 GiB 2.6 TiB 71.52 1.09 127     up 
 66   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  52 KiB  15 GiB 3.5 TiB 61.53 0.94 107     up 
 67   hdd 9.09470  1.00000 9.1 TiB 5.4 TiB 5.3 TiB  52 KiB  15 GiB 3.7 TiB 58.88 0.90 105     up 
 84   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB  16 GiB 3.3 TiB 63.58 0.97 114     up 
 85   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  20 KiB  19 GiB 2.2 TiB 75.81 1.15 135     up 
 86   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  18 GiB 2.7 TiB 70.31 1.07 128     up 
 87   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.6 TiB  44 KiB  18 GiB 2.4 TiB 73.16 1.11 130     up 
 88   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB  17 GiB 2.8 TiB 69.07 1.05 125     up 
 89   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  16 GiB 3.0 TiB 66.83 1.02 122     up 
 90   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  84 KiB  17 GiB 2.9 TiB 68.15 1.04 120     up 
 91   hdd 9.09470  1.00000 9.1 TiB 5.4 TiB 5.3 TiB  48 KiB  15 GiB 3.7 TiB 58.97 0.90 104     up 
 76   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 100 KiB  16 GiB 3.3 TiB 63.62 0.97 115     up 
 77   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  48 KiB  16 GiB 3.4 TiB 62.08 0.94 110     up 
 78   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB  16 GiB 3.3 TiB 64.12 0.98 115     up 
 79   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  84 KiB  16 GiB 3.3 TiB 63.57 0.97 115     up 
 80   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB  68 KiB  19 GiB 2.1 TiB 76.64 1.17 138     up 
 81   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.1 TiB  49 KiB  16 GiB 2.9 TiB 67.64 1.03 121     up 
 82   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  36 KiB  17 GiB 2.9 TiB 68.03 1.03 120     up 
 83   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  68 KiB  17 GiB 2.8 TiB 69.21 1.05 121     up 
 68   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  32 KiB  18 GiB 2.3 TiB 74.22 1.13 133     up 
 69   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.2 TiB  72 KiB  17 GiB 2.8 TiB 68.81 1.05 123     up 
 70   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  36 KiB  17 GiB 2.9 TiB 68.15 1.04 123     up 
 71   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 128 KiB  16 GiB 3.2 TiB 64.29 0.98 114     up 
 72   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 5.9 TiB  76 KiB  16 GiB 3.1 TiB 65.56 1.00 120     up 
 73   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  76 KiB  16 GiB 3.3 TiB 64.18 0.98 118     up 
 74   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  92 KiB  17 GiB 2.6 TiB 71.68 1.09 131     up 
 75   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB 116 KiB  15 GiB 3.5 TiB 61.78 0.94 112     up 
180   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  16 GiB 3.2 TiB 64.73 0.98 115     up 
186   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  16 GiB 2.9 TiB 68.51 1.04 124     up 
192   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  72 KiB  17 GiB 2.9 TiB 68.41 1.04 124     up 
197   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  56 KiB  15 GiB 3.3 TiB 63.70 0.97 114     up 
202   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 104 KiB  16 GiB 3.2 TiB 65.07 0.99 116     up 
208   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.8 TiB  32 KiB  18 GiB 2.2 TiB 75.45 1.15 133     up 
213   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 6.3 MiB  16 GiB 3.3 TiB 64.02 0.97 117     up 
219   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  92 KiB  15 GiB 3.2 TiB 65.16 0.99 118     up 
176   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  88 KiB  16 GiB 3.3 TiB 63.89 0.97 115     up 
182   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  17 GiB 2.8 TiB 69.52 1.06 127     up 
188   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  17 GiB 2.6 TiB 71.13 1.08 127     up 
196   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  48 KiB  17 GiB 2.9 TiB 68.14 1.04 122     up 
201   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  52 KiB  15 GiB 3.4 TiB 62.63 0.95 114     up 
207   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB 104 KiB  17 GiB 2.6 TiB 71.05 1.08 131     up 
214   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB  16 GiB 2.8 TiB 69.53 1.06 125     up 
218   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  60 KiB  16 GiB 3.2 TiB 64.81 0.99 119     up 
179   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  84 KiB  15 GiB 3.1 TiB 65.64 1.00 119     up 
185   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.0 TiB 132 KiB  16 GiB 3.0 TiB 66.54 1.01 119     up 
191   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  17 GiB 2.9 TiB 68.46 1.04 122     up 
198   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  16 GiB 3.1 TiB 65.92 1.00 121     up 
203   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB 108 KiB  18 GiB 2.2 TiB 75.79 1.15 134     up 
209   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  36 KiB  17 GiB 3.1 TiB 65.78 1.00 120     up 
215   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  52 KiB  16 GiB 3.1 TiB 65.68 1.00 120     up 
221   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.6 TiB  68 KiB  15 GiB 3.4 TiB 62.19 0.95 111     up 
177   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  32 KiB  18 GiB 2.6 TiB 71.44 1.09 130     up 
183   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  88 KiB  18 GiB 2.2 TiB 76.23 1.16 141     up 
190   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  52 KiB  18 GiB 2.3 TiB 74.20 1.13 133     up 
194   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  16 GiB 3.1 TiB 65.70 1.00 122     up 
199   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB 104 KiB  17 GiB 2.7 TiB 70.36 1.07 122     up 
205   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  48 KiB  16 GiB 3.1 TiB 66.22 1.01 120     up 
211   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  88 KiB  17 GiB 2.3 TiB 74.83 1.14 137     up 
217   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  48 KiB  17 GiB 2.6 TiB 71.71 1.09 130     up 
178   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB  15 GiB 3.4 TiB 62.70 0.95 115     up 
184   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  16 GiB 3.2 TiB 64.77 0.99 119     up 
189   hdd 9.09470  1.00000 9.1 TiB 4.9 TiB 4.9 TiB  92 KiB  13 GiB 4.2 TiB 54.30 0.83  98     up 
195   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  36 KiB  16 GiB 3.1 TiB 66.27 1.01 118     up 
200   hdd 9.09470  1.00000 9.1 TiB 5.3 TiB 5.3 TiB  56 KiB  15 GiB 3.8 TiB 57.95 0.88 103     up 
206   hdd 9.09470  1.00000 9.1 TiB 5.5 TiB 5.5 TiB  80 KiB  15 GiB 3.6 TiB 60.59 0.92 109     up 
212   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.1 TiB 132 KiB  16 GiB 2.9 TiB 67.71 1.03 124     up 
216   hdd 9.09470  1.00000 9.1 TiB 5.1 TiB 5.1 TiB  56 KiB  14 GiB 4.0 TiB 56.28 0.86  99     up 
136   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB  15 GiB 3.3 TiB 64.08 0.97 114     up 
137   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  52 KiB  16 GiB 3.2 TiB 65.23 0.99 118     up 
138   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.5 TiB  92 KiB  17 GiB 2.5 TiB 72.17 1.10 132     up 
139   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  56 KiB  17 GiB 2.8 TiB 69.18 1.05 125     up 
140   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 108 KiB  16 GiB 3.2 TiB 64.86 0.99 115     up 
141   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB  16 GiB 3.0 TiB 66.72 1.02 121     up 
142   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  88 KiB  18 GiB 2.2 TiB 76.21 1.16 138     up 
143   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB 100 KiB  16 GiB 3.2 TiB 64.96 0.99 116     up 
121   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  32 KiB  17 GiB 2.8 TiB 68.97 1.05 121     up 
123   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB 136 KiB  19 GiB 2.4 TiB 74.10 1.13 131     up 
124   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 128 KiB  16 GiB 3.0 TiB 67.48 1.03 121     up 
126   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  20 KiB  18 GiB 2.8 TiB 69.55 1.06 125     up 
128   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  96 KiB  16 GiB 3.0 TiB 66.99 1.02 120     up 
130   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  24 KiB  18 GiB 2.9 TiB 68.49 1.04 125     up 
132   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  40 KiB  17 GiB 2.9 TiB 68.11 1.04 122     up 
134   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  80 KiB  16 GiB 3.5 TiB 61.26 0.93 106     up 
146   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.2 TiB  64 KiB  16 GiB 2.8 TiB 68.86 1.05 124     up 
149   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  96 KiB  18 GiB 2.4 TiB 73.83 1.12 129     up 
152   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  28 KiB  17 GiB 2.8 TiB 68.94 1.05 124     up 
155   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  17 GiB 2.7 TiB 70.56 1.07 128     up 
158   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB  52 KiB  17 GiB 2.6 TiB 70.92 1.08 129     up 
161   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  56 KiB  16 GiB 3.0 TiB 66.87 1.02 122     up 
164   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB  52 KiB  17 GiB 2.6 TiB 71.05 1.08 127     up 
167   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 108 KiB  15 GiB 3.4 TiB 63.07 0.96 117     up 
144   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  16 GiB 3.0 TiB 67.00 1.02 120     up 
147   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  93 KiB  16 GiB 2.8 TiB 69.41 1.06 123     up 
150   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  96 KiB  16 GiB 2.8 TiB 69.12 1.05 125     up 
153   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  40 KiB  17 GiB 2.8 TiB 68.67 1.04 125     up 
156   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  48 KiB  18 GiB 2.7 TiB 70.15 1.07 128     up 
159   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  60 KiB  16 GiB 3.0 TiB 67.03 1.02 119     up 
163   hdd 9.09470  1.00000 9.1 TiB 7.0 TiB 6.9 TiB  56 KiB  18 GiB 2.1 TiB 76.58 1.16 138     up 
166   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  88 KiB  16 GiB 3.1 TiB 65.72 1.00 120     up 
145   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  16 GiB 2.7 TiB 70.00 1.06 127     up 
148   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 124 KiB  16 GiB 2.9 TiB 68.34 1.04 124     up 
151   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  80 KiB  16 GiB 3.0 TiB 67.09 1.02 118     up 
154   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  40 KiB  16 GiB 3.0 TiB 66.74 1.02 121     up 
157   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  92 KiB  17 GiB 3.0 TiB 66.90 1.02 116     up 
160   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  84 KiB  17 GiB 2.9 TiB 68.53 1.04 125     up 
162   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  76 KiB  17 GiB 2.6 TiB 71.80 1.09 129     up 
165   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  48 KiB  17 GiB 2.4 TiB 73.32 1.12 131     up 
112   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  17 GiB 2.9 TiB 68.16 1.04 123     up 
113   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  88 KiB  16 GiB 3.2 TiB 65.04 0.99 115     up 
114   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  64 KiB  16 GiB 3.2 TiB 64.73 0.98 117     up 
115   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  72 KiB  17 GiB 3.0 TiB 67.26 1.02 119     up 
116   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  24 KiB  16 GiB 3.3 TiB 63.48 0.97 115     up 
117   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.8 TiB  84 KiB  19 GiB 2.2 TiB 75.42 1.15 138     up 
118   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  92 KiB  17 GiB 2.8 TiB 69.45 1.06 124     up 
119   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  15 GiB 3.5 TiB 61.80 0.94 112     up 
120   hdd 9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  92 KiB  16 GiB 3.3 TiB 63.40 0.96 116     up 
122   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.0 TiB  52 KiB  16 GiB 3.0 TiB 66.59 1.01 118     up 
125   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 120 KiB  16 GiB 3.4 TiB 62.73 0.95 114     up 
127   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  44 KiB  18 GiB 2.6 TiB 71.58 1.09 129     up 
129   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  17 GiB 2.7 TiB 70.06 1.07 125     up 
131   hdd 9.09470  1.00000 9.1 TiB 5.5 TiB 5.4 TiB 108 KiB  15 GiB 3.6 TiB 59.98 0.91 108     up 
133   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  64 KiB  15 GiB 3.5 TiB 61.28 0.93 109     up 
135   hdd 9.09470  1.00000 9.1 TiB 5.6 TiB 5.6 TiB  56 KiB  15 GiB 3.5 TiB 61.80 0.94 113     up 
168   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  16 GiB 2.9 TiB 67.94 1.03 124     up 
169   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  17 GiB 2.8 TiB 69.43 1.06 125     up 
170   hdd 9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  48 KiB  15 GiB 3.4 TiB 62.67 0.95 111     up 
171   hdd 9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  44 KiB  15 GiB 3.2 TiB 65.12 0.99 119     up 
172   hdd 9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  32 KiB  16 GiB 3.0 TiB 67.19 1.02 121     up 
173   hdd 9.09470  1.00000 9.1 TiB 5.4 TiB 5.4 TiB  84 KiB  14 GiB 3.7 TiB 59.32 0.90 104     up 
174   hdd 9.09470  1.00000 9.1 TiB 5.5 TiB 5.5 TiB  56 KiB  15 GiB 3.6 TiB 60.40 0.92 109     up 
175   hdd 9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  84 KiB  17 GiB 2.7 TiB 70.61 1.07 121     up 
181   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  44 KiB  18 GiB 2.4 TiB 73.69 1.12 134     up 
187   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  88 KiB  16 GiB 2.8 TiB 69.02 1.05 122     up 
193   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  64 KiB  17 GiB 2.5 TiB 72.77 1.11 133     up 
204   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  52 KiB  16 GiB 2.9 TiB 68.53 1.04 124     up 
210   hdd 9.09470  1.00000 9.1 TiB 6.9 TiB 6.9 TiB 100 KiB  18 GiB 2.2 TiB 76.29 1.16 138     up 
220   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  48 KiB  17 GiB 2.6 TiB 71.89 1.09 133     up 
222   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  96 KiB  17 GiB 2.5 TiB 72.35 1.10 132     up 
223   hdd 9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB  17 GiB 2.5 TiB 72.55 1.10 131     up 
 92   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  72 KiB  17 GiB 2.9 TiB 67.94 1.03 123     up 
 93   hdd 9.09470  1.00000 9.1 TiB 6.5 TiB 6.4 TiB  84 KiB  17 GiB 2.6 TiB 71.01 1.08 129     up 
 94   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  17 GiB 2.8 TiB 69.07 1.05 123     up 
 95   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  80 KiB  16 GiB 3.1 TiB 65.81 1.00 120     up 
 96   hdd 9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  16 GiB 2.9 TiB 68.07 1.04 121     up 
 97   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  52 KiB  17 GiB 2.8 TiB 69.25 1.05 127     up 
 98   hdd 9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB 124 KiB  17 GiB 2.8 TiB 69.55 1.06 126     up 
 99   hdd 9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  84 KiB  16 GiB 3.1 TiB 66.04 1.00 120     up

There's 3 OSD with > 80% and one at almost 90 whilst others are in the low 60s.
Upmade PG Balancer mode. Someone on the ML said there are fixes introduced soon? There is some major bug there

#6 Updated by super xor 6 months ago

...
 19   hdd 9.09470  1.00000 9.1 TiB 6.8 TiB  6.8 TiB  92 KiB  18 GiB 2.2 TiB 75.28 1.07 132     up 
 22   hdd 9.09470  1.00000 9.1 TiB 7.1 TiB  7.1 TiB  88 KiB  19 GiB 2.0 TiB 78.20 1.11 137     up 
 25   hdd 9.09470  1.00000 9.1 TiB 8.0 TiB  8.0 TiB  11 MiB  21 GiB 1.1 TiB 87.80 1.25 155     up 
 30   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB  6.7 TiB  60 KiB  18 GiB 2.3 TiB 74.20 1.05 130     up 
 33   hdd 9.09470  1.00000 9.1 TiB 6.7 TiB  6.7 TiB  80 KiB  18 GiB 2.4 TiB 74.12 1.05 130     up 
...

OSD 25 nearfull, so I reweight to 0.9 and what happens?

 25   hdd 9.09470  0.89999 9.1 TiB 8.0 TiB  8.0 TiB  11 MiB  21 GiB 1.1 TiB 87.87 1.25 155     up

It's moving data ONTO the OSD, not OFF.

Also available in: Atom PDF