Ceph » mgr

This problem went away as I created additional pools, but has now resurfaced in v14.2.8. The output of 'ceph df' alternates between sometimes being correct (roughly 10% of the time):

[root@ceph ~]# cat strace17.out
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.3 PiB 252 TiB 1.1 PiB 1.1 PiB 81.43
mdd 2.0 TiB 2.0 TiB 21 GiB 30 GiB 1.47
ssd 257 TiB 255 TiB 1.9 TiB 2.3 TiB 0.91
TOTAL 1.6 PiB 509 TiB 1.1 PiB 1.1 PiB 68.54

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
fs.metadata.archive 35 383 MiB 4.22k 646 MiB 0.03 637 GiB
fs.data.archive 36 0 B 566.29k 0 B 0 81 TiB
fs.data.archive.frames 38 835 TiB 211.04M 1.1 PiB 88.35 109 TiB
fs.metadata.scratch 44 1.1 GiB 1.47k 1.7 GiB 0.09 637 GiB
fs.data.scratch 45 0 B 2.07M 0 B 0 81 TiB
fs.data.scratch.llcache 46 814 GiB 2.08M 1019 GiB 0.41 181 TiB
fs.data.scratch.frames 47 0 B 0 0 B 0 181 TiB
fs.metadata.home 48 287 MiB 70 649 MiB 0.03 637 GiB
fs.data.home 49 2 KiB 2 48 KiB 0 81 TiB
fs.data.home.user 50 384 KiB 1 288 KiB 0 181 TiB

but mostly incorrect:

[root@ceph ~]# cat strace16.out
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.3 PiB 263 TiB 1.1 PiB 1.1 PiB 80.78
mdd 2.0 TiB 2.0 TiB 21 GiB 30 GiB 1.47
ssd 257 TiB 255 TiB 1.9 TiB 2.3 TiB 0.91
TOTAL 1.6 PiB 520 TiB 1.1 PiB 1.1 PiB 68.08

POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL
fs.metadata.archive 35 383 MiB 4.22k 383 MiB 0.02 637 GiB
fs.data.archive 36 0 B 566.29k 0 B 0 81 TiB
fs.data.archive.frames 38 804 TiB 211.04M 804 TiB 84.68 109 TiB
fs.metadata.scratch 44 1.1 GiB 1.47k 1.1 GiB 0.06 637 GiB
fs.data.scratch 45 0 B 2.07M 0 B 0 81 TiB
fs.data.scratch.llcache 46 667 GiB 2.08M 667 GiB 0.27 181 TiB
fs.data.scratch.frames 47 0 B 0 0 B 0 181 TiB
fs.metadata.home 48 287 MiB 70 287 MiB 0.01 637 GiB
fs.data.home 49 2 KiB 2 2 KiB 0 81 TiB
fs.data.home.user 50 2 B 1 2 B 0 181 TiB

Note there are discrepancies even in the STORED column, with Ceph sometimes indicating that 384KiB are stored in the fs.data.home.user pool (a 6+2 EC pool), but generally reporting the 2-byte size of the solitary file currently in that pool. I suppose "2 B" is the correct STORED value in this case, but I am fairly certain that file is USING more than 2 bytes of the pool. In fact, I am trying to empirically determine just how much data and metadata space files of different sizes take up, and this is seriously hampering that effort. Some of the above pools are triply-replicated, others are erasure-coded:

[root@ceph ~]# ceph osd pool ls detail
pool 35 'fs.metadata.archive' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 52155 flags hashpspool stripe_width 0 target_size_ratio 0.1 application cephfs
pool 36 'fs.data.archive' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 47695 flags hashpspool stripe_width 0 compression_algorithm lz4 compression_mode aggressive target_size_ratio 0.01 application cephfs
pool 38 'fs.data.archive.frames' erasure size 8 min_size 7 crush_rule 2 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 141214 lfor 0/0/80364 flags hashpspool,ec_overwrites stripe_width 393216 compression_algorithm lz4 compression_mode aggressive target_size_ratio 1 application cephfs
pool 44 'fs.metadata.scratch' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 58512 lfor 0/58512/58510 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 target_size_ratio 0.05 application cephfs
pool 45 'fs.data.scratch' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 47654 flags hashpspool stripe_width 0 compression_algorithm lz4 compression_mode aggressive target_size_ratio 0.01 application cephfs
pool 46 'fs.data.scratch.llcache' erasure size 8 min_size 7 crush_rule 4 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 93444 lfor 0/80331/80329 flags hashpspool,ec_overwrites stripe_width 393216 compression_algorithm lz4 compression_mode aggressive target_size_ratio 0.05 application cephfs
pool 47 'fs.data.scratch.frames' erasure size 8 min_size 7 crush_rule 5 object_hash rjenkins pg_num 8192 pgp_num 8192 autoscale_mode warn last_change 120429 lfor 0/0/92563 flags hashpspool,ec_overwrites stripe_width 393216 application cephfs
pool 48 'fs.metadata.home' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 130182 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
pool 49 'fs.data.home' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 130180 flags hashpspool stripe_width 0 compression_algorithm lz4 compression_mode aggressive application cephfs
pool 50 'fs.data.home.user' erasure size 8 min_size 7 crush_rule 6 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 130181 flags hashpspool,ec_overwrites stripe_width 393216 compression_algorithm lz4 compression_mode aggressive application cephfs

strace16.out and strace17.out shown above are what 'strace ceph df' displayed on STDOUT. The corresponding STDERR streams are attached.