Project

General

Profile

Bug #35946

ceph df strange usage

Added by Yunzhi Cheng over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

 cluster:
    id:     10f9c55a-a813-44d7-bce7-e6159a98dc61
    health: HEALTH_WARN
            1 nearfull osd(s)
            2 pool(s) nearfull

  services:
    mon: 3 daemons, quorum rndcl94,rndcl106,rndcl154
    mgr: rndcl94(active), standbys: rndcl106
    mds: cephfs-1/1/1 up  {0=rndcl94=up:active}, 1 up:standby
    osd: 18 osds: 18 up, 18 in

  data:
    pools:   2 pools, 512 pgs
    objects: 71.98 M objects, 14 TiB
    usage:   50 TiB used, 16 TiB / 67 TiB avail
    pgs:     509 active+clean
             3   active+clean+scrubbing+deep

  io:
    client:   319 MiB/s rd, 19 KiB/s wr, 2.23 kop/s rd, 1 op/s wr
ceph df detail
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED     OBJECTS
    67 TiB     16 TiB       50 TiB         75.42     71.98 M
POOLS:
    NAME                ID     QUOTA OBJECTS     QUOTA BYTES     USED        %USED     MAX AVAIL     OBJECTS      DIRTY       READ        WRITE       RAW USED
    cephfs_data         1      N/A               N/A              14 TiB     90.01       1.5 TiB     71303564     71.30 M      93 MiB     205 MiB       41 TiB
    cephfs_metadata     2      N/A               N/A             5.4 GiB      0.35       1.5 TiB       680937     680.9 k     3.8 MiB      13 MiB       16 GiB
ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
 1   hdd 3.70679  0.95001 3.7 TiB 2.9 TiB 780 GiB 79.46 1.05  82
 5   hdd 3.70679  1.00000 3.7 TiB 2.6 TiB 1.1 TiB 70.78 0.94  91
 8   hdd 3.70679  1.00000 3.7 TiB 3.1 TiB 651 GiB 82.84 1.10  98
11   hdd 3.70679  1.00000 3.7 TiB 2.9 TiB 847 GiB 77.69 1.03  85
14   hdd 3.70679  1.00000 3.7 TiB 2.4 TiB 1.3 TiB 65.80 0.87  76
16   hdd 3.70679  1.00000 3.7 TiB 2.8 TiB 912 GiB 75.96 1.01  80
 2   hdd 3.70679  1.00000 3.7 TiB 2.8 TiB 913 GiB 75.94 1.01  78
 4   hdd 3.70679  1.00000 3.7 TiB 3.0 TiB 715 GiB 81.16 1.08  79
 7   hdd 3.70679  1.00000 3.7 TiB 2.2 TiB 1.5 TiB 60.51 0.80  75
10   hdd 3.70679  1.00000 3.7 TiB 2.9 TiB 849 GiB 77.64 1.03  95
15   hdd 3.70679  1.00000 3.7 TiB 2.8 TiB 973 GiB 74.36 0.99  83
17   hdd 3.70679  1.00000 3.7 TiB 3.1 TiB 648 GiB 82.93 1.10 102
 0   hdd 3.70679  0.85004 3.7 TiB 3.3 TiB 449 GiB 88.17 1.17  95
 3   hdd 3.70679  1.00000 3.7 TiB 2.7 TiB 1.0 TiB 72.56 0.96  82
 6   hdd 3.70679  0.95001 3.7 TiB 3.1 TiB 585 GiB 84.58 1.12  86
 9   hdd 3.70679  1.00000 3.7 TiB 2.5 TiB 1.2 TiB 67.37 0.89  86
12   hdd 3.70679  1.00000 3.7 TiB 2.6 TiB 1.1 TiB 69.05 0.92  76
13   hdd 3.70679  1.00000 3.7 TiB 2.6 TiB 1.1 TiB 70.79 0.94  87
                    TOTAL  67 TiB  50 TiB  16 TiB 75.42
MIN/MAX VAR: 0.80/1.17  STDDEV: 6.96

I see cephfs_data raw_usage is 41T and cephfs_metadata raw_usage is 16G. but why global raw_usage is 50T ?

and why global raw_usage is 75.42% but cephfs_data usage is 90% ?

History

#1 Updated by Sage Weil over 5 years ago

  • Status changed from New to Closed

The global usage is just adding the used and total stats (statfs(2) output for filestore) for every OSD in the system. Used there will include metadata overhead, anything else on the file system, etc.

The USED% is based on MAX AVAIL, which is how much can be written into that pool, which is bounded by the most-full OSD that participates in the pool.. so it will generally be different than the overall available free bytes in the total system.

#2 Updated by Sage Weil over 5 years ago

As it happens we are improving the internal metrics we are collecting so these values will be (hopefully) less confusing in Nautilus. See https://github.com/ceph/ceph/pull/19454

Also available in: Atom PDF