Ceph » Linux kernel client

Hello,

I'm currently putting a small ceph cluster under pressure. After a while, I finally figured out why my used space stats where wrong.

My issue is evident with a simple 'ceph df detail':

GLOBAL:
    SIZE     AVAIL     RAW USED     %RAW USED     OBJECTS 
    258T      253T        4646G          1.76        131k 
POOLS:
    NAME                 ID     QUOTA OBJECTS     QUOTA BYTES     USED      %USED     MAX AVAIL     OBJECTS     DIRTY     READ      WRITE     RAW USED 
    cephfs_data_21p3     5      N/A               N/A             1936G      0.87          215T      118298      115k     48377     6965k        2213G 
    cephfs_metadata      6      N/A               N/A              413M         0        84198G       16687     16687     55252      642k        1133M

=> The raw used for global and pool completely mismatch

Without finding a real hint in the doc about it, I put the max object size for cephfs at ~1GB (with setfattr) to reduce the number of objects (my files are huge).

After quite a lot of digging, I tried to extract some large objects with 'rados get': they where of size 128MB instead of 1GB.
Then I found that: https://github.com/ceph/ceph/pull/15520, and with a little calculation get that the 'RAW USED' for pool is based on max-128MB objects (but they are larger in reality). This calculation get exactly what is displayed in df, so I'm quite confident that it is the source of that issue.

Reading the files with cephfs is not a problem, it is "just" a stat issue.

I suggest to either put the 128MB limitation into setfattr (and by the way producing basic error msgs for that command, and maybe document that limit ?), or to correct the stats in df.

Thanks !