Ceph

In our setup we have two rulesets, one for SSDs and another one for HDDs. Ceph df normally reports the MAX AVAIL space considering the OSDs in the ruleset, but when on of the OSDs is down and out it just reports 0 instead of the real MAX AVAIL space for the pools using that ruleset.

Command "ceph fs" output:

GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
28309G 22750G 5558G 19.63 757k
POOLS:
NAME ID CATEGORY USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE
ssd-volumes-01 118 - 5344M 0.02 0 1472 1472 4861k 5281k
hdd-volumes-01 120 - 0 0 8200G 0 0 0 0

ssd-volumes-01 pool is using ruleset 0 (It has one OSD down and out)
hdd-volumes-01 pool is using ruleset 1 (All OSDs are up)