Ceph » RADOS

The pool df stats are supposed to preset user bytes as "stored" and raw used as "bytes_used".

But if any osd is up with reweight 0.0 (i.e. out) then the bytes_used is == stored, without the factor for replication or EC.

E.g. with 2 osds up but out (as shown from osd df tree, see https://termbin.com/9mx1):

 100   hdd   10.91600        0     0 B     0 B     0 B     0 B     0 B     0 B     0    0   0     up                     osd.100          
 177   hdd   10.91600        0     0 B     0 B     0 B     0 B     0 B     0 B     0    0   0     up                     osd.177          

Here are the stats with those osds out:

# ceph df
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED 
    hdd       5.5 PiB     1.2 PiB     4.3 PiB      4.3 PiB         78.46 
    TOTAL     5.5 PiB     1.2 PiB     4.3 PiB      4.3 PiB         78.46 

POOLS:
    POOL       ID     STORED      OBJECTS     USED        %USED     MAX AVAIL 
    public     68     2.9 PiB     143.56M     2.9 PiB     78.48       538 TiB 
    test       71      29 MiB       6.56k      29 MiB         0       269 TiB 
    foo        72     1.2 GiB         308     1.2 GiB         0       269 TiB 

Then I simply stopped the two out osds, and now the stats are correct:

# ceph df
RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED 
    hdd       5.5 PiB     1.2 PiB     4.3 PiB      4.3 PiB         78.44 
    TOTAL     5.5 PiB     1.2 PiB     4.3 PiB      4.3 PiB         78.44 

POOLS:
    POOL       ID     STORED      OBJECTS     USED        %USED     MAX AVAIL 
    public     68     2.9 PiB     143.32M     4.3 PiB     84.36       544 TiB 
    test       71      29 MiB       6.59k     1.2 GiB         0       272 TiB 
    foo        72     1.2 GiB         308     3.6 GiB         0       272 TiB 

I fixed another affected cluster the same way -- found an up but out OSD, and stopped it.