Bug #35946
ceph df strange usage
0%
Description
cluster:
id: 10f9c55a-a813-44d7-bce7-e6159a98dc61
health: HEALTH_WARN
1 nearfull osd(s)
2 pool(s) nearfull
services:
mon: 3 daemons, quorum rndcl94,rndcl106,rndcl154
mgr: rndcl94(active), standbys: rndcl106
mds: cephfs-1/1/1 up {0=rndcl94=up:active}, 1 up:standby
osd: 18 osds: 18 up, 18 in
data:
pools: 2 pools, 512 pgs
objects: 71.98 M objects, 14 TiB
usage: 50 TiB used, 16 TiB / 67 TiB avail
pgs: 509 active+clean
3 active+clean+scrubbing+deep
io:
client: 319 MiB/s rd, 19 KiB/s wr, 2.23 kop/s rd, 1 op/s wr
ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
67 TiB 16 TiB 50 TiB 75.42 71.98 M
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED
cephfs_data 1 N/A N/A 14 TiB 90.01 1.5 TiB 71303564 71.30 M 93 MiB 205 MiB 41 TiB
cephfs_metadata 2 N/A N/A 5.4 GiB 0.35 1.5 TiB 680937 680.9 k 3.8 MiB 13 MiB 16 GiB
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
1 hdd 3.70679 0.95001 3.7 TiB 2.9 TiB 780 GiB 79.46 1.05 82
5 hdd 3.70679 1.00000 3.7 TiB 2.6 TiB 1.1 TiB 70.78 0.94 91
8 hdd 3.70679 1.00000 3.7 TiB 3.1 TiB 651 GiB 82.84 1.10 98
11 hdd 3.70679 1.00000 3.7 TiB 2.9 TiB 847 GiB 77.69 1.03 85
14 hdd 3.70679 1.00000 3.7 TiB 2.4 TiB 1.3 TiB 65.80 0.87 76
16 hdd 3.70679 1.00000 3.7 TiB 2.8 TiB 912 GiB 75.96 1.01 80
2 hdd 3.70679 1.00000 3.7 TiB 2.8 TiB 913 GiB 75.94 1.01 78
4 hdd 3.70679 1.00000 3.7 TiB 3.0 TiB 715 GiB 81.16 1.08 79
7 hdd 3.70679 1.00000 3.7 TiB 2.2 TiB 1.5 TiB 60.51 0.80 75
10 hdd 3.70679 1.00000 3.7 TiB 2.9 TiB 849 GiB 77.64 1.03 95
15 hdd 3.70679 1.00000 3.7 TiB 2.8 TiB 973 GiB 74.36 0.99 83
17 hdd 3.70679 1.00000 3.7 TiB 3.1 TiB 648 GiB 82.93 1.10 102
0 hdd 3.70679 0.85004 3.7 TiB 3.3 TiB 449 GiB 88.17 1.17 95
3 hdd 3.70679 1.00000 3.7 TiB 2.7 TiB 1.0 TiB 72.56 0.96 82
6 hdd 3.70679 0.95001 3.7 TiB 3.1 TiB 585 GiB 84.58 1.12 86
9 hdd 3.70679 1.00000 3.7 TiB 2.5 TiB 1.2 TiB 67.37 0.89 86
12 hdd 3.70679 1.00000 3.7 TiB 2.6 TiB 1.1 TiB 69.05 0.92 76
13 hdd 3.70679 1.00000 3.7 TiB 2.6 TiB 1.1 TiB 70.79 0.94 87
TOTAL 67 TiB 50 TiB 16 TiB 75.42
MIN/MAX VAR: 0.80/1.17 STDDEV: 6.96
I see cephfs_data raw_usage is 41T and cephfs_metadata raw_usage is 16G. but why global raw_usage is 50T ?
and why global raw_usage is 75.42% but cephfs_data usage is 90% ?
History
#1 Updated by Sage Weil over 5 years ago
- Status changed from New to Closed
The global usage is just adding the used and total stats (statfs(2) output for filestore) for every OSD in the system. Used there will include metadata overhead, anything else on the file system, etc.
The USED% is based on MAX AVAIL, which is how much can be written into that pool, which is bounded by the most-full OSD that participates in the pool.. so it will generally be different than the overall available free bytes in the total system.
#2 Updated by Sage Weil over 5 years ago
As it happens we are improving the internal metrics we are collecting so these values will be (hopefully) less confusing in Nautilus. See https://github.com/ceph/ceph/pull/19454