Project

General

Profile

Bug #63215

bluestore/rocksdb: bluefs_db_used_bytes is inconsistent with DB stats

Added by yite gu 4 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

1. db usage from osd df

$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP      META     AVAIL    %USE   VAR   PGS  STATUS
 1   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    10 KiB   94 GiB  1.1 TiB  84.61  1.00   99      up
 4   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    17 KiB   89 GiB  1.1 TiB  84.55  1.00  100      up
 8   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    30 KiB   60 GiB  860 GiB  87.97  1.04  103      up
11   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    13 KiB  100 GiB  1.1 TiB  84.67  1.00  101      up
14   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    25 KiB   56 GiB  858 GiB  88.00  1.04  100      up
17   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    26 KiB   94 GiB  1.1 TiB  84.64  1.00   99      up
20   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    13 MiB   74 GiB  976 GiB  86.35  1.02  100      up
32   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.6 TiB    17 KiB  146 GiB  1.2 TiB  82.33  0.97  100      up
 0   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    11 KiB   26 GiB  1.1 TiB  84.61  1.00    0    down
 3   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    32 KiB   65 GiB  996 GiB  86.08  1.01  100      up
 6   nvme  6.98630   1.00000  7.0 TiB  6.4 TiB  6.3 TiB    29 KiB   53 GiB  651 GiB  90.90  1.07  100      up
 9   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    16 KiB  106 GiB  1.1 TiB  83.88  0.99  101      up
12   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.6 TiB    29 KiB  155 GiB  1.2 TiB  82.54  0.97  100      up
15   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  5.9 TiB     7 KiB   65 GiB  1.0 TiB  85.17  1.00  100      up
18   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  5.9 TiB     6 KiB   94 GiB  1.0 TiB  85.49  1.01   99      up
30   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    25 KiB   87 GiB  1.1 TiB  84.51  0.99  101      up
 2   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB    15 KiB  133 GiB  1.2 TiB  83.15  0.98  100      up
 5   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB   996 KiB   83 GiB  1.1 TiB  84.41  0.99  101      up
 7   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.0 TiB    22 KiB   68 GiB  929 GiB  87.02  1.02  101      up
10   nvme  6.98630   1.00000  7.0 TiB  6.3 TiB  6.3 TiB     8 KiB   50 GiB  656 GiB  90.83  1.07   99      up
13   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  5.9 TiB    16 KiB   63 GiB  1.0 TiB  85.21  1.00  100      up
16   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.6 TiB    35 KiB  141 GiB  1.2 TiB  82.26  0.97  101      up
19   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.5 TiB     8 KiB  146 GiB  1.4 TiB  80.39  0.95  100      up
31   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB  1004 KiB  131 GiB  1.2 TiB  83.20  0.98  100      up
23   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    14 KiB  108 GiB  1.1 TiB  84.74  1.00  100      up
35   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    14 KiB   90 GiB  1.1 TiB  84.52  1.00   99      up
44   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB   980 KiB  135 GiB  1.0 TiB  85.10  1.00  100      up
53   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  5.9 TiB    27 KiB  107 GiB  1.0 TiB  85.65  1.01  101      up
62   nvme  6.98630   1.00000  7.0 TiB  5.3 TiB  5.2 TiB  1007 KiB  145 GiB  1.7 TiB  76.28  0.90  100      up
71   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB    12 KiB  114 GiB  1.2 TiB  83.00  0.98   99      up
80   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    19 KiB   74 GiB  985 GiB  86.23  1.02  100      up
89   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    13 MiB   65 GiB  1.0 TiB  85.15  1.00   99      up
25   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    16 KiB   87 GiB  1.2 TiB  83.53  0.98  100      up
36   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.6 TiB     8 KiB  168 GiB  1.2 TiB  82.71  0.97   99      up
46   nvme  6.98630   1.00000  7.0 TiB  6.4 TiB  6.4 TiB    19 KiB   53 GiB  591 GiB  91.73  1.08  101      up
55   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.6 TiB    13 KiB  160 GiB  1.2 TiB  82.57  0.97   99      up
64   nvme  6.98630   1.00000  7.0 TiB  6.2 TiB  6.1 TiB    30 KiB   62 GiB  855 GiB  88.05  1.04  100      up
73   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  5.9 TiB    20 KiB   73 GiB  1.0 TiB  85.29  1.00   99      up
82   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    30 KiB  105 GiB  1.1 TiB  84.56  1.00  101      up
91   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    11 KiB   84 GiB  968 GiB  86.47  1.02  100      up
22   nvme  6.98630   1.00000  7.0 TiB  6.2 TiB  6.2 TiB    22 KiB   56 GiB  797 GiB  88.87  1.05  100      up
34   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    36 KiB   62 GiB  861 GiB  87.97  1.04  103      up
43   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB     8 KiB   76 GiB  1.1 TiB  84.45  0.99  100      up
52   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    20 KiB   80 GiB  1.1 TiB  84.32  0.99  100      up
61   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB    21 KiB   98 GiB  1.1 TiB  83.66  0.98   99      up
70   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.6 TiB    13 KiB  188 GiB  1.2 TiB  82.12  0.97   98      up
79   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.0 TiB    33 KiB   64 GiB  926 GiB  87.05  1.02  100      up
88   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.4 TiB    28 KiB  203 GiB  1.4 TiB  80.17  0.94  100      up
                       TOTAL  335 TiB  285 TiB  280 TiB    30 MiB  4.5 TiB   51 TiB  84.94

2. db usage from rocksdb stats
** Compaction Stats [O-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   214.38 MB   0.8      0.0     0.0      0.0    1316.1   1316.1       0.0   1.0      0.0    309.8   4349.85           3919.75      6458    0.674       0      0
  L1      5/0   231.91 MB   0.9   2004.6  1315.9    688.7    1753.5   1064.9       0.0   1.3    182.7    159.8  11234.96           8209.95      3228    3.480   3040M   462M
  L2     42/0    2.50 GB   1.0   5483.0  1064.1   4418.9    4767.0    348.1       0.6   4.5    147.6    128.3  38039.57          23427.71     17591    2.162   7641M  1079M
  L3   1018/0    4.91 GB   0.2   1010.6   346.2    664.4     665.4      0.9       0.0   1.9    162.9    107.3   6351.13           3622.79      5528    1.149   1390M   469M
 Sum   1066/0    7.84 GB   0.0   8498.2  2726.2   5772.0    8502.1   2730.1       0.6   6.5    145.1    145.2  59975.51          39180.20     32805    1.828     12G  2012M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

History

#1 Updated by yite gu 4 months ago

** Compaction Stats [O-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   214.74 MB   0.8      0.0     0.0      0.0    1317.7   1317.7       0.0   1.0      0.0    306.8   4397.41           4005.50      6458    0.681       0      0
  L1      3/0   195.85 MB   0.8   2007.3  1317.5    689.8    1755.8   1066.0       0.0   1.3    184.5    161.4  11142.79           7896.38      3228    3.452   3043M   463M
  L2     41/0    2.49 GB   1.0   5488.0  1065.3   4422.7    4772.1    349.4       0.6   4.5    147.0    127.8  38235.92          23532.56     17678    2.163   7651M  1079M
  L3   1024/0    4.92 GB   0.2   1016.9   347.4    669.5     670.4      0.9       0.0   1.9    162.2    107.0   6418.61           3644.93      5543    1.158   1398M   471M
 Sum   1069/0    7.81 GB   0.0   8512.2  2730.2   5782.0    8516.0   2734.0       0.6   6.5    144.8    144.9  60194.74          39079.37     32907    1.829     12G  2014M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [O-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   213.99 MB   0.8      0.0     0.0      0.0    1316.4   1316.4       0.0   1.0      0.0    312.1   4319.34           3870.24      6458    0.669       0      0
  L1      4/0   216.89 MB   0.8   2004.5  1316.2    688.2    1753.6   1065.4       0.0   1.3    180.9    158.2  11348.35           8324.00      3228    3.516   3040M   462M
  L2     43/0    2.50 GB   1.0   5491.7  1064.5   4427.2    4775.4    348.2       0.7   4.5    148.3    129.0  37912.04          23344.51     17612    2.153   7651M  1079M
  L3    733/0    4.91 GB   0.2   1011.0   346.4    664.6     665.6      0.9       0.0   1.9    163.6    107.7   6326.79           3624.22      5531    1.144   1393M   470M
 Sum    781/0    7.83 GB   0.0   8507.2  2727.1   5780.1    8511.0   2730.9       0.7   6.5    145.4    145.5  59906.52          39162.98     32829    1.825     12G  2012M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

#2 Updated by yite gu 4 months ago

yite gu wrote:

[...]
[...]

from osd.88

#3 Updated by Igor Fedotov 4 months ago

@yite gu - You probably should account for all the column families - doing that for 'O' one is not enough

#4 Updated by yite gu 4 months ago

Igor Fedotov wrote:

@yite gu - You probably should account for all the column families - doing that for 'O' one is not enough

I know, but other column families usage very small:

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0   62.07 MB   0.8      0.0     0.0      0.0     135.4    135.4       0.0   1.0      0.0     40.7   3408.05           3348.83      6458    0.528       0      0
  L1      4/0   217.11 MB   0.8    481.5   135.3    346.2     393.3     47.0       0.0   2.9     39.9     32.6  12361.22          10655.83      1614    7.659     15G  2776M
  L2     22/0   468.99 MB   0.2    139.6    46.8     92.8      92.8      0.0       0.0   2.0     54.5     36.2   2625.04           2320.71       694    3.782   4465M  1497M
 Sum     29/0   748.17 MB   0.0    621.2   182.1    439.0     621.4    182.4       0.0   4.6     34.6     34.6  18394.31          16325.36      8766    2.098     19G  4273M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0    3.11 KB   0.8      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      2.3      0.01              0.01        17    0.000       0      0
  L1      4/0    7.04 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   0.0     49.9      0.0      0.00              0.00         2    0.000      50     50
 Sum      7/0   10.15 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      1.6      2.3      0.01              0.01        19    0.000      50     50
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      3/0    3.11 KB   0.8      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      1.8      0.01              0.00        12    0.001       0      0
  L1     11/0   11.98 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum     14/0   15.09 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      1.8      0.01              0.00        12    0.001       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0    1.18 KB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      2.3      0.01              0.00        13    0.000       0      0
  L1      1/0    2.50 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0     43.6      0.0      0.00              0.00         7    0.000      44     44
 Sum      2/0    3.68 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      2.0      2.2      0.01              0.00        20    0.000      44     44
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [L] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      2/0    1.96 KB   0.5      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      4.1      0.17              0.13       605    0.000       0      0
  L1     48/0   256.34 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   0.2     40.1      3.4      0.01              0.00        37    0.000     109    104
 Sum     50/0   258.30 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.1      2.8      4.1      0.19              0.14       642    0.000     109    104
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [P] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0     537.0    537.0       0.0   1.0      0.0    121.7   4517.07           4323.23      8162    0.553       0      0
  L1      2/0   81.54 MB   0.3   1186.1   537.0    649.1     649.1      0.0       0.0   1.2    167.5     91.7   7251.22           5317.44      8161    0.889   9513M  5514M
 Sum      2/0   81.54 MB   0.0   1186.1   537.0    649.1    1186.1    537.0       0.0   2.2    103.2    103.2  11768.28           9640.68     16323    0.721   9513M  5514M
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

#5 Updated by Igor Fedotov 4 months ago

hmm... I presume RocksDB numbers are inaccurate. Perhaps it doesn't take removed data records into account or something.
As a brute force investigation approach I'd suggest to export bluefs via ceph-bluestore-tool's bluefs-export command and learn the actual size it takes. Also you might try to compact the DB and check what would be the stats afterwards...

#6 Updated by yite gu 4 months ago

Igor Fedotov wrote:

hmm... I presume RocksDB numbers are inaccurate. Perhaps it doesn't take removed data records into account or something.
As a brute force investigation approach I'd suggest to export bluefs via ceph-bluestore-tool's bluefs-export command and learn the actual size it takes. Also you might try to compact the DB and check what would be the stats afterwards...

1. I compacted all DB, the stats as below:
osd df:

$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP      META     AVAIL    %USE   VAR   PGS  STATUS
 1   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    10 KiB   18 GiB  1.1 TiB  83.54  1.00   99      up
 4   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    13 KiB   18 GiB  1.1 TiB  83.56  1.00  100      up
 8   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    13 KiB   19 GiB  902 GiB  87.39  1.04  103      up
11   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    10 KiB   18 GiB  1.2 TiB  83.53  1.00  101      up
14   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    11 KiB   19 GiB  895 GiB  87.48  1.04  100      up
17   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    18 KiB   18 GiB  1.1 TiB  83.58  1.00   99      up
20   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    13 MiB   19 GiB  1.0 TiB  85.58  1.02  100      up
32   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB    17 KiB   17 GiB  1.4 TiB  80.54  0.96  100      up
 0   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    12 KiB   18 GiB  1.1 TiB  84.50  1.01  100      up
 3   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    28 KiB   18 GiB  1.0 TiB  85.43  1.02  100      up
 6   nvme  6.98630   1.00000  7.0 TiB  6.3 TiB  6.3 TiB    29 KiB   20 GiB  685 GiB  90.43  1.08  100      up
 9   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    13 KiB   18 GiB  1.2 TiB  82.65  0.99  101      up
12   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB    24 KiB   17 GiB  1.4 TiB  80.62  0.96  100      up
15   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB     7 KiB   18 GiB  1.1 TiB  84.51  1.01  100      up
18   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB     6 KiB   18 GiB  1.1 TiB  84.43  1.01   99      up
30   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    15 KiB   18 GiB  1.1 TiB  83.55  1.00  101      up
 2   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB    15 KiB   18 GiB  1.3 TiB  81.54  0.97  100      up
 5   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   982 KiB   18 GiB  1.2 TiB  83.51  1.00  101      up
 7   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    23 KiB   19 GiB  978 GiB  86.33  1.03  101      up
10   nvme  6.98630   1.00000  7.0 TiB  6.3 TiB  6.3 TiB     8 KiB   20 GiB  686 GiB  90.41  1.08   99      up
13   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    16 KiB   18 GiB  1.1 TiB  84.58  1.01  100      up
16   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB    20 KiB   17 GiB  1.4 TiB  80.54  0.96  101      up
19   nvme  6.98630   1.00000  7.0 TiB  5.5 TiB  5.5 TiB     8 KiB   17 GiB  1.5 TiB  78.58  0.94  100      up
31   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB  1000 KiB   18 GiB  1.3 TiB  81.61  0.97  100      up
23   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    15 KiB   18 GiB  1.2 TiB  83.49  1.00  100      up
35   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    14 KiB   18 GiB  1.2 TiB  83.52  1.00   99      up
44   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   986 KiB   18 GiB  1.2 TiB  83.46  1.00  100      up
53   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    14 KiB   18 GiB  1.1 TiB  84.41  1.01  101      up
62   nvme  6.98630   1.00000  7.0 TiB  5.2 TiB  5.2 TiB  1000 KiB   16 GiB  1.8 TiB  74.49  0.89  100      up
71   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB    15 KiB   18 GiB  1.3 TiB  81.66  0.97   99      up
80   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    12 KiB   18 GiB  1.0 TiB  85.46  1.02  100      up
89   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    13 MiB   18 GiB  1.1 TiB  84.50  1.01   99      up
25   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    16 KiB   18 GiB  1.2 TiB  82.56  0.98  100      up
36   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB     8 KiB   18 GiB  1.4 TiB  80.60  0.96   99      up
46   nvme  6.98630   1.00000  7.0 TiB  6.4 TiB  6.4 TiB    14 KiB   23 GiB  621 GiB  91.32  1.09  101      up
55   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB     9 KiB   17 GiB  1.4 TiB  80.57  0.96   99      up
64   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    15 KiB   19 GiB  898 GiB  87.45  1.04  100      up
73   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    12 KiB   18 GiB  1.1 TiB  84.53  1.01   99      up
82   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    24 KiB   18 GiB  1.2 TiB  83.33  0.99  101      up
91   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    11 KiB   19 GiB  1.0 TiB  85.56  1.02  100      up
22   nvme  6.98630   1.00000  7.0 TiB  6.2 TiB  6.2 TiB    17 KiB   19 GiB  833 GiB  88.35  1.05  100      up
34   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB    29 KiB   19 GiB  904 GiB  87.37  1.04  103      up
43   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB     8 KiB   18 GiB  1.1 TiB  83.64  1.00  100      up
52   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB    12 KiB   18 GiB  1.2 TiB  83.46  1.00  100      up
61   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB    10 KiB   18 GiB  1.2 TiB  82.54  0.98   99      up
70   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB     9 KiB   17 GiB  1.4 TiB  79.73  0.95   98      up
79   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB    22 KiB   19 GiB  972 GiB  86.42  1.03  100      up
88   nvme  6.98630   1.00000  7.0 TiB  5.4 TiB  5.4 TiB    20 KiB   18 GiB  1.6 TiB  77.57  0.93  100      up
                       TOTAL  335 TiB  281 TiB  280 TiB    30 MiB  875 GiB   54 TiB  83.84

osd.88 perf dump:

    "bluefs": {
        "db_total_bytes": 7681501126656,
        "db_used_bytes": 18912141312,
        "wal_total_bytes": 0,
        "wal_used_bytes": 0,
        "slow_total_bytes": 0,
        "slow_used_bytes": 0,
        "num_files": 313,
        "log_bytes": 4935680,
        "log_compactions": 2,
        "logged_bytes": 30945280,
        "files_written_wal": 5,
        "files_written_sst": 506,
        "bytes_written_wal": 38068224,
        "bytes_written_sst": 32528220160,
        "bytes_written_slow": 0,
        "max_bytes_wal": 0,
        "max_bytes_db": 219581239296,
        "max_bytes_slow": 0,
        "read_random_count": 89446,
        "read_random_bytes": 43336365348,
        "read_random_disk_count": 70823,
        "read_random_disk_bytes": 43104689672,
        "read_random_buffer_count": 55903,
        "read_random_buffer_bytes": 231675676,
        "read_count": 25533,
        "read_bytes": 1047650237,
        "read_prefetch_count": 7694,
        "read_prefetch_bytes": 479221816,
$ ceph tell osd.88 bluestore bluefs device info
{
    "dev": {
        "device": "BDEV_DB",
        "total": 7681501126656,
        "free": 1723064070144,
        "bluefs_used": 18912141312,
        "bluefs max available": 1723064070144
    }
}

Everything seems normal now.

2. I run performance test now. I would bluefs-export after reproduct. It may take 1 day.

#7 Updated by yite gu 4 months ago

By the way, we no this problem at v16.2.13.

#8 Updated by yite gu 4 months ago

osd.88 rocksdb stats now:

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0     58.4      0.12              0.00         2    0.062       0      0
  L1      0/0    0.00 KB   0.0      0.5     0.1      0.5       0.5      0.0       0.0   7.3     57.1     53.6      9.45              8.35         2    4.724     16M  1009K
  L2      7/0   468.96 MB   0.2      0.7     0.2      0.5       0.5     -0.0       0.0   1.9     68.8     44.7     10.50              9.19         1   10.502     22M  7905K
 Sum      7/0   468.96 MB   0.0      1.2     0.3      0.9       1.0      0.0       0.0 136.3     62.9     49.0     20.07             17.54         5    4.015     39M  8915K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [m-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sum      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      1.5      0.00              0.00         1    0.001       0      0
  L1      8/0   11.20 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum      8/0   11.20 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      1.5      0.00              0.00         1    0.001       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L1      5/0    5.68 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   0.0     36.8      0.0      0.00              0.00         1    0.000      33     33
 Sum      5/0    5.68 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   0.0     36.8      0.0      0.00              0.00         1    0.000      33     33
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [p-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L1      2/0    3.68 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum      2/0    3.68 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [O-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.1      0.1       0.0   1.0      0.0    478.3      0.16              0.00         2    0.079       0      0
  L1      0/0    0.00 KB   0.0      0.7     0.3      0.4       0.7      0.2       0.0   2.4    134.4    127.6      5.50              3.63         2    2.751   1084K    68K
  L2      0/0    0.00 KB   0.0      3.7     0.4      3.3       3.4      0.2       0.0   7.8    134.5    125.0     28.20             19.03         4    7.049   5109K   410K
  L3     77/0    4.92 GB   0.2      7.9     2.7      5.2       5.2      0.0       0.0   2.0    156.6    103.7     51.57             35.81         4   12.894     10M  3637K
 Sum     77/0    4.92 GB   0.0     12.3     3.4      8.9       9.4      0.5       0.0 127.3    147.6    112.9     85.43             58.47        12    7.119     17M  4116K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [O-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.1      0.1       0.0   1.0      0.0    555.7      0.14              0.00         2    0.068       0      0
  L1      0/0    0.00 KB   0.0      0.7     0.3      0.4       0.7      0.2       0.0   2.3    127.5    119.7      5.63              3.66         2    2.814   1051K    81K
  L2      0/0    0.00 KB   0.0      4.2     0.5      3.7       3.9      0.2       0.0   8.4    146.3    136.4     29.25             20.06         6    4.876   5724K   439K
  L3     77/0    4.91 GB   0.2      7.8     2.7      5.2       5.2      0.0       0.0   1.9    154.9    102.0     51.90             35.92         4   12.975     10M  3634K
 Sum     77/0    4.91 GB   0.0     12.7     3.4      9.3       9.8      0.5       0.0 133.1    150.0    115.4     86.92             59.64        14    6.208     17M  4155K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [O-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.1      0.1       0.0   1.0      0.0    555.5      0.14              0.00         2    0.068       0      0
  L1      0/0    0.00 KB   0.0      0.7     0.3      0.4       0.7      0.2       0.0   2.4    133.6    126.2      5.51              3.60         2    2.756   1076K    75K
  L2      0/0    0.00 KB   0.0      4.0     0.5      3.6       3.7      0.2       0.0   8.2    112.8    104.8     36.48             25.06         5    7.295   5490K   438K
  L3     77/0    4.92 GB   0.2      7.9     2.7      5.2       5.2      0.0       0.0   1.9    133.3     88.1     60.47             43.37         4   15.117     10M  3614K
 Sum     77/0    4.92 GB   0.0     12.6     3.4      9.2       9.7      0.5       0.0 132.1    125.8     96.7    102.59             72.03        13    7.892     17M  4129K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [L] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0      2.9      0.00              0.00         2    0.002       0      0
  L1     46/0   230.13 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   0.0    689.8      0.0      0.00              0.00         1    0.000       8      8
 Sum     46/0   230.13 KB   0.0      0.0     0.0      0.0       0.0     -0.0       0.0   1.0     11.0      2.9      0.00              0.00         3    0.001       8      8
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

** Compaction Stats [P] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   1.0      0.0    269.5      0.13              0.01         2    0.064       0      0
  L1      2/0   81.45 MB   0.3      0.2     0.0      0.2       0.2     -0.0       0.0   4.7    373.1    307.4      0.53              0.52         2    0.265   1247K   267K
 Sum      2/0   81.45 MB   0.0      0.2     0.0      0.2       0.2      0.0       0.0   5.7    300.1    300.0      0.66              0.53         4    0.165   1247K   267K
 Int      0/0    0.00 KB   0.0      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0

#9 Updated by yite gu 4 months ago

performance test is already running 33 hours. osd df as below:

[rook@n049-079-175-212 /]$ ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL     %USE   VAR   PGS  STATUS
 1   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   15 KiB   27 GiB   1.1 TiB  83.68  1.00   99      up
 4   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   16 KiB   27 GiB   1.1 TiB  83.69  1.00  100      up
 8   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB   25 KiB   28 GiB   893 GiB  87.52  1.04  103      up
11   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   15 KiB   28 GiB   1.1 TiB  83.66  1.00  101      up
14   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB   19 KiB   30 GiB   884 GiB  87.64  1.04  100      up
17   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   18 KiB   28 GiB   1.1 TiB  83.72  1.00   99      up
20   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   13 MiB   31 GiB  1019 GiB  85.75  1.02  100      up
32   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB   26 KiB   27 GiB   1.3 TiB  80.68  0.96  100      up
 0   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB   18 KiB   28 GiB   1.1 TiB  84.63  1.01  100      up
 3   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   29 KiB   28 GiB   1.0 TiB  85.56  1.02  100      up
 6   nvme  6.98630   1.00000  7.0 TiB  6.3 TiB  6.3 TiB   29 KiB   31 GiB   673 GiB  90.59  1.08  100      up
 9   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   16 KiB   30 GiB   1.2 TiB  82.82  0.99  101      up
12   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB   42 KiB   27 GiB   1.3 TiB  80.75  0.96  100      up
15   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    7 KiB   29 GiB   1.1 TiB  84.66  1.01  100      up
18   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB    6 KiB   29 GiB   1.1 TiB  84.58  1.01   99      up
30   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   21 KiB   28 GiB   1.1 TiB  83.69  1.00  101      up
 2   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB   21 KiB   26 GiB   1.3 TiB  81.66  0.97  100      up
 5   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB  1.0 MiB   28 GiB   1.1 TiB  83.65  1.00  101      up
 7   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   38 KiB   29 GiB   968 GiB  86.47  1.03  101      up
10   nvme  6.98630   1.00000  7.0 TiB  6.3 TiB  6.3 TiB    8 KiB   29 GiB   676 GiB  90.55  1.08   99      up
13   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB   16 KiB   31 GiB   1.1 TiB  84.76  1.01  100      up
16   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB   28 KiB   28 GiB   1.3 TiB  80.69  0.96  101      up
19   nvme  6.98630   1.00000  7.0 TiB  5.5 TiB  5.5 TiB    8 KiB   26 GiB   1.5 TiB  78.71  0.94  100      up
31   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB  1.0 MiB   27 GiB   1.3 TiB  81.75  0.97  100      up
23   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   17 KiB   27 GiB   1.1 TiB  83.61  1.00  100      up
35   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   21 KiB   28 GiB   1.1 TiB  83.66  1.00   99      up
44   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB  1.0 MiB   34 GiB   1.1 TiB  83.68  1.00  100      up
53   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB   19 KiB   29 GiB   1.1 TiB  84.55  1.01  101      up
62   nvme  6.98630   1.00000  7.0 TiB  5.2 TiB  5.2 TiB  1.0 MiB   26 GiB   1.8 TiB  74.63  0.89  100      up
71   nvme  6.98630   1.00000  7.0 TiB  5.7 TiB  5.7 TiB   23 KiB   28 GiB   1.3 TiB  81.81  0.97   99      up
80   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   17 KiB   29 GiB   1.0 TiB  85.60  1.02  100      up
89   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB   13 MiB   29 GiB   1.1 TiB  84.66  1.01   99      up
25   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   17 KiB   32 GiB   1.2 TiB  82.75  0.99  100      up
36   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB    8 KiB   26 GiB   1.3 TiB  80.72  0.96   99      up
46   nvme  6.98630   1.00000  7.0 TiB  6.4 TiB  6.4 TiB   18 KiB   34 GiB   610 GiB  91.47  1.09  101      up
55   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB    9 KiB   29 GiB   1.3 TiB  80.73  0.96   99      up
64   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB   23 KiB   31 GiB   885 GiB  87.62  1.04  100      up
73   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.9 TiB   16 KiB   29 GiB   1.1 TiB  84.68  1.01   99      up
82   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   29 KiB   28 GiB   1.2 TiB  83.48  0.99  101      up
91   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   11 KiB   28 GiB  1023 GiB  85.70  1.02  100      up
22   nvme  6.98630   1.00000  7.0 TiB  6.2 TiB  6.2 TiB   27 KiB   29 GiB   823 GiB  88.49  1.05  100      up
34   nvme  6.98630   1.00000  7.0 TiB  6.1 TiB  6.1 TiB   36 KiB   28 GiB   895 GiB  87.49  1.04  103      up
43   nvme  6.98630   1.00000  7.0 TiB  5.9 TiB  5.8 TiB    8 KiB   30 GiB   1.1 TiB  83.81  1.00  100      up
52   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.8 TiB   20 KiB   29 GiB   1.1 TiB  83.61  1.00  100      up
61   nvme  6.98630   1.00000  7.0 TiB  5.8 TiB  5.7 TiB   12 KiB   28 GiB   1.2 TiB  82.68  0.98   99      up
70   nvme  6.98630   1.00000  7.0 TiB  5.6 TiB  5.6 TiB   12 KiB   27 GiB   1.4 TiB  79.86  0.95   98      up
79   nvme  6.98630   1.00000  7.0 TiB  6.0 TiB  6.0 TiB   24 KiB   28 GiB   962 GiB  86.55  1.03  100      up
88   nvme  6.98630   1.00000  7.0 TiB  5.4 TiB  5.4 TiB   28 KiB   27 GiB   1.6 TiB  77.70  0.93  100      up
                       TOTAL  335 TiB  282 TiB  280 TiB   31 MiB  1.3 TiB    54 TiB  83.99
MIN/MAX VAR: 0.89/1.09  STDDEV: 3.19

osd.88 perf dump
    "bluefs": {
        "db_total_bytes": 7681501126656,
        "db_used_bytes": 28729425920,
        "wal_total_bytes": 0,
        "wal_used_bytes": 0,
        "slow_total_bytes": 0,
        "slow_used_bytes": 0,
        "num_files": 442,
        "log_bytes": 12300288,
        "log_compactions": 88258,
        "logged_bytes": 1462521335808,
        "files_written_wal": 11944,
        "files_written_sst": 109582,
        "bytes_written_wal": 3913517834240,
        "bytes_written_sst": 7575278858240,
        "bytes_written_slow": 0,
        "max_bytes_wal": 0,
        "max_bytes_db": 219581239296,
        "max_bytes_slow": 0,
        "read_random_count": 908406827,
        "read_random_bytes": 11193391440067,
        "read_random_disk_count": 913483859,
        "read_random_disk_bytes": 11187406700651,
        "read_random_buffer_count": 478355,
        "read_random_buffer_bytes": 5984739416,
        "read_count": 134607,
        "read_bytes": 48021127527,
        "read_prefetch_count": 116768,
        "read_prefetch_bytes": 47452699106,

$ ceph tell osd.88 bluestore bluefs device info
{
    "dev": {
        "device": "BDEV_DB",
        "total": 7681501126656,
        "free": 1713107165184,
        "bluefs_used": 28858605568,
        "bluefs max available": 1713107165184
    }
}

There have been no osd of high db usage.

#10 Updated by yite gu 4 months ago

now, I pick osd.20 to export bluefs.
1. db usage 31G from osd.20 meta stats
2. db usage from rocksdb stats:

** Compaction Stats [default] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      0/0    0.00 KB   0.0      0.0     0.0      0.0      58.4     58.4       0.0   1.0      0.0     39.5   1515.69           1484.22      2536    0.598       0      0
  L1      3/0   201.06 MB   0.8    194.5    58.4    136.0     158.2     22.2       0.0   2.7     37.3     30.4   5336.97           4584.73       634    8.418   6197M  1140M
  L2     11/0   468.98 MB   0.2     65.4    22.0     43.5      43.5      0.0       0.0   2.0     50.1     33.3   1337.17           1183.99       325    4.114   2092M   702M
 Sum     14/0   670.04 MB   0.0    259.9    80.4    179.5     260.1     80.6       0.0   4.5     32.5     32.5   8189.83           7252.94      3495    2.343   8289M  1842M
 Int      0/0    0.00 KB   0.0      0.8     0.2      0.6       0.8      0.2       0.0   5.5     35.8     34.2     23.43             20.48        10    2.343     26M  5652K

** Compaction Stats [O-0] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   197.62 MB   0.8      0.0     0.0      0.0     482.9    482.9       0.0   1.0      0.0    287.3   1721.14           1553.06      2675    0.643       0      0
  L1      4/0   242.43 MB   0.9    756.5   482.7    273.8     620.0    346.2       0.0   1.3    202.2    165.7   3832.22           2843.17      1267    3.025   1078M   234M
  L2     47/0    1.95 GB   0.8   1737.7   345.6   1392.1    1393.7      1.6       0.4   4.0    148.4    119.0  11990.00           7653.32      5610    2.137   2302M   462M
  L3     86/0    5.45 GB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum    138/0    7.83 GB   0.0   2494.3   828.3   1665.9    2496.6    830.7       0.4   5.2    145.6    145.7  17543.35          12049.54      9552    1.837   3381M   697M
 Int      0/0    0.00 KB   0.0      6.8     2.2      4.7       6.9      2.2       0.0   5.1    149.2    150.1     46.83             31.43        26    1.801   9226K  1881K

** Compaction Stats [O-1] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   196.04 MB   0.8      0.0     0.0      0.0     479.9    479.9       0.0   1.0      0.0    294.9   1666.34           1494.36      2536    0.657       0      0
  L1      4/0   202.04 MB   0.8    746.0   479.8    266.3     612.6    346.3       0.0   1.3    204.4    167.9   3736.66           2868.35      1267    2.949   1067M   229M
  L2     44/0    1.94 GB   0.8   1742.3   345.7   1396.6    1398.1      1.6       0.4   4.0    146.4    117.5  12184.41           7701.77      5603    2.175   2313M   463M
  L3     85/0    5.45 GB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum    134/0    7.78 GB   0.0   2488.3   825.5   1662.8    2490.6    827.8       0.4   5.2    144.9    145.0  17587.41          12064.49      9406    1.870   3381M   693M
 Int      0/0    0.00 KB   0.0      6.7     2.1      4.6       6.8      2.2       0.0   5.0    150.1    150.6     46.03             30.89        26    1.770   9120K  1895K

** Compaction Stats [O-2] **
Level    Files   Size     Score Read(GB)  Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  L0      1/0   197.13 MB   0.8      0.0     0.0      0.0     480.2    480.2       0.0   1.0      0.0    297.2   1654.84           1477.12      2536    0.653       0      0
  L1      4/0   242.95 MB   0.9    752.3   480.0    272.2     616.6    344.4       0.0   1.3    199.6    163.6   3859.15           3015.89      1267    3.046   1072M   233M
  L2     45/0    1.94 GB   0.8   1730.9   343.7   1387.2    1388.7      1.6       0.4   4.0    147.9    118.7  11982.22           7587.17      5612    2.135   2295M   460M
  L3     85/0    5.44 GB   0.2      0.0     0.0      0.0       0.0      0.0       0.0   0.0      0.0      0.0      0.00              0.00         0    0.000       0      0
 Sum    135/0    7.81 GB   0.0   2483.1   823.8   1659.4    2485.5    826.2       0.4   5.2    145.3    145.5  17496.22          12080.18      9415    1.858   3368M   693M
 Int      0/0    0.00 KB   0.0      7.1     2.2      4.8       7.0      2.2       0.0   5.2    151.4    150.9     47.68             32.39        27    1.766   9523K  1960K

other column families usage very small, we could ignore.
670MB + 7.83 GB + 7.78 GB + 7.81 GB = 24.07GB
3. bluefs-export
# ceph-bluestore-tool bluefs-export --path /var/lib/ceph/osd/ceph-20 --out-dir ./osd-20-db-export

# du -h ./osd-20-db-export/
8.0K    ./osd-20-db-export/sharding
4.0K    ./osd-20-db-export/db.slow
25G    ./osd-20-db-export/db
255M    ./osd-20-db-export/db.wal
25G    ./osd-20-db-export/

Also available in: Atom PDF