Actions
Bug #63215
openbluestore/rocksdb: bluefs_db_used_bytes is inconsistent with DB stats
Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
1. db usage from osd df
$ ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 10 KiB 94 GiB 1.1 TiB 84.61 1.00 99 up
4 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 17 KiB 89 GiB 1.1 TiB 84.55 1.00 100 up
8 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 30 KiB 60 GiB 860 GiB 87.97 1.04 103 up
11 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 13 KiB 100 GiB 1.1 TiB 84.67 1.00 101 up
14 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 25 KiB 56 GiB 858 GiB 88.00 1.04 100 up
17 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 26 KiB 94 GiB 1.1 TiB 84.64 1.00 99 up
20 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 13 MiB 74 GiB 976 GiB 86.35 1.02 100 up
32 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.6 TiB 17 KiB 146 GiB 1.2 TiB 82.33 0.97 100 up
0 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 11 KiB 26 GiB 1.1 TiB 84.61 1.00 0 down
3 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 32 KiB 65 GiB 996 GiB 86.08 1.01 100 up
6 nvme 6.98630 1.00000 7.0 TiB 6.4 TiB 6.3 TiB 29 KiB 53 GiB 651 GiB 90.90 1.07 100 up
9 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 16 KiB 106 GiB 1.1 TiB 83.88 0.99 101 up
12 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.6 TiB 29 KiB 155 GiB 1.2 TiB 82.54 0.97 100 up
15 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 5.9 TiB 7 KiB 65 GiB 1.0 TiB 85.17 1.00 100 up
18 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 5.9 TiB 6 KiB 94 GiB 1.0 TiB 85.49 1.01 99 up
30 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 25 KiB 87 GiB 1.1 TiB 84.51 0.99 101 up
2 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 15 KiB 133 GiB 1.2 TiB 83.15 0.98 100 up
5 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 996 KiB 83 GiB 1.1 TiB 84.41 0.99 101 up
7 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.0 TiB 22 KiB 68 GiB 929 GiB 87.02 1.02 101 up
10 nvme 6.98630 1.00000 7.0 TiB 6.3 TiB 6.3 TiB 8 KiB 50 GiB 656 GiB 90.83 1.07 99 up
13 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 5.9 TiB 16 KiB 63 GiB 1.0 TiB 85.21 1.00 100 up
16 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.6 TiB 35 KiB 141 GiB 1.2 TiB 82.26 0.97 101 up
19 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.5 TiB 8 KiB 146 GiB 1.4 TiB 80.39 0.95 100 up
31 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 1004 KiB 131 GiB 1.2 TiB 83.20 0.98 100 up
23 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 14 KiB 108 GiB 1.1 TiB 84.74 1.00 100 up
35 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 14 KiB 90 GiB 1.1 TiB 84.52 1.00 99 up
44 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 980 KiB 135 GiB 1.0 TiB 85.10 1.00 100 up
53 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 5.9 TiB 27 KiB 107 GiB 1.0 TiB 85.65 1.01 101 up
62 nvme 6.98630 1.00000 7.0 TiB 5.3 TiB 5.2 TiB 1007 KiB 145 GiB 1.7 TiB 76.28 0.90 100 up
71 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 12 KiB 114 GiB 1.2 TiB 83.00 0.98 99 up
80 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 19 KiB 74 GiB 985 GiB 86.23 1.02 100 up
89 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 13 MiB 65 GiB 1.0 TiB 85.15 1.00 99 up
25 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 16 KiB 87 GiB 1.2 TiB 83.53 0.98 100 up
36 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.6 TiB 8 KiB 168 GiB 1.2 TiB 82.71 0.97 99 up
46 nvme 6.98630 1.00000 7.0 TiB 6.4 TiB 6.4 TiB 19 KiB 53 GiB 591 GiB 91.73 1.08 101 up
55 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.6 TiB 13 KiB 160 GiB 1.2 TiB 82.57 0.97 99 up
64 nvme 6.98630 1.00000 7.0 TiB 6.2 TiB 6.1 TiB 30 KiB 62 GiB 855 GiB 88.05 1.04 100 up
73 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 5.9 TiB 20 KiB 73 GiB 1.0 TiB 85.29 1.00 99 up
82 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 30 KiB 105 GiB 1.1 TiB 84.56 1.00 101 up
91 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 11 KiB 84 GiB 968 GiB 86.47 1.02 100 up
22 nvme 6.98630 1.00000 7.0 TiB 6.2 TiB 6.2 TiB 22 KiB 56 GiB 797 GiB 88.87 1.05 100 up
34 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 36 KiB 62 GiB 861 GiB 87.97 1.04 103 up
43 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 8 KiB 76 GiB 1.1 TiB 84.45 0.99 100 up
52 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 20 KiB 80 GiB 1.1 TiB 84.32 0.99 100 up
61 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 21 KiB 98 GiB 1.1 TiB 83.66 0.98 99 up
70 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.6 TiB 13 KiB 188 GiB 1.2 TiB 82.12 0.97 98 up
79 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.0 TiB 33 KiB 64 GiB 926 GiB 87.05 1.02 100 up
88 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.4 TiB 28 KiB 203 GiB 1.4 TiB 80.17 0.94 100 up
TOTAL 335 TiB 285 TiB 280 TiB 30 MiB 4.5 TiB 51 TiB 84.94
2. db usage from rocksdb stats
** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 214.38 MB 0.8 0.0 0.0 0.0 1316.1 1316.1 0.0 1.0 0.0 309.8 4349.85 3919.75 6458 0.674 0 0 L1 5/0 231.91 MB 0.9 2004.6 1315.9 688.7 1753.5 1064.9 0.0 1.3 182.7 159.8 11234.96 8209.95 3228 3.480 3040M 462M L2 42/0 2.50 GB 1.0 5483.0 1064.1 4418.9 4767.0 348.1 0.6 4.5 147.6 128.3 38039.57 23427.71 17591 2.162 7641M 1079M L3 1018/0 4.91 GB 0.2 1010.6 346.2 664.4 665.4 0.9 0.0 1.9 162.9 107.3 6351.13 3622.79 5528 1.149 1390M 469M Sum 1066/0 7.84 GB 0.0 8498.2 2726.2 5772.0 8502.1 2730.1 0.6 6.5 145.1 145.2 59975.51 39180.20 32805 1.828 12G 2012M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Updated by yite gu 7 months ago
** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 214.74 MB 0.8 0.0 0.0 0.0 1317.7 1317.7 0.0 1.0 0.0 306.8 4397.41 4005.50 6458 0.681 0 0 L1 3/0 195.85 MB 0.8 2007.3 1317.5 689.8 1755.8 1066.0 0.0 1.3 184.5 161.4 11142.79 7896.38 3228 3.452 3043M 463M L2 41/0 2.49 GB 1.0 5488.0 1065.3 4422.7 4772.1 349.4 0.6 4.5 147.0 127.8 38235.92 23532.56 17678 2.163 7651M 1079M L3 1024/0 4.92 GB 0.2 1016.9 347.4 669.5 670.4 0.9 0.0 1.9 162.2 107.0 6418.61 3644.93 5543 1.158 1398M 471M Sum 1069/0 7.81 GB 0.0 8512.2 2730.2 5782.0 8516.0 2734.0 0.6 6.5 144.8 144.9 60194.74 39079.37 32907 1.829 12G 2014M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 213.99 MB 0.8 0.0 0.0 0.0 1316.4 1316.4 0.0 1.0 0.0 312.1 4319.34 3870.24 6458 0.669 0 0 L1 4/0 216.89 MB 0.8 2004.5 1316.2 688.2 1753.6 1065.4 0.0 1.3 180.9 158.2 11348.35 8324.00 3228 3.516 3040M 462M L2 43/0 2.50 GB 1.0 5491.7 1064.5 4427.2 4775.4 348.2 0.7 4.5 148.3 129.0 37912.04 23344.51 17612 2.153 7651M 1079M L3 733/0 4.91 GB 0.2 1011.0 346.4 664.6 665.6 0.9 0.0 1.9 163.6 107.7 6326.79 3624.22 5531 1.144 1393M 470M Sum 781/0 7.83 GB 0.0 8507.2 2727.1 5780.1 8511.0 2730.9 0.7 6.5 145.4 145.5 59906.52 39162.98 32829 1.825 12G 2012M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Updated by Igor Fedotov 7 months ago
@yite gu - You probably should account for all the column families - doing that for 'O' one is not enough
Updated by yite gu 7 months ago
Igor Fedotov wrote:
@yite gu - You probably should account for all the column families - doing that for 'O' one is not enough
I know, but other column families usage very small:
** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 62.07 MB 0.8 0.0 0.0 0.0 135.4 135.4 0.0 1.0 0.0 40.7 3408.05 3348.83 6458 0.528 0 0 L1 4/0 217.11 MB 0.8 481.5 135.3 346.2 393.3 47.0 0.0 2.9 39.9 32.6 12361.22 10655.83 1614 7.659 15G 2776M L2 22/0 468.99 MB 0.2 139.6 46.8 92.8 92.8 0.0 0.0 2.0 54.5 36.2 2625.04 2320.71 694 3.782 4465M 1497M Sum 29/0 748.17 MB 0.0 621.2 182.1 439.0 621.4 182.4 0.0 4.6 34.6 34.6 18394.31 16325.36 8766 2.098 19G 4273M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.11 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.01 0.01 17 0.000 0 0 L1 4/0 7.04 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 49.9 0.0 0.00 0.00 2 0.000 50 50 Sum 7/0 10.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.6 2.3 0.01 0.01 19 0.000 50 50 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 3.11 KB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.01 0.00 12 0.001 0 0 L1 11/0 11.98 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 14/0 15.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.8 0.01 0.00 12 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 1.18 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.01 0.00 13 0.000 0 0 L1 1/0 2.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 43.6 0.0 0.00 0.00 7 0.000 44 44 Sum 2/0 3.68 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 2.0 2.2 0.01 0.00 20 0.000 44 44 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 1.96 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.17 0.13 605 0.000 0 0 L1 48/0 256.34 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.2 40.1 3.4 0.01 0.00 37 0.000 109 104 Sum 50/0 258.30 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.1 2.8 4.1 0.19 0.14 642 0.000 109 104 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 537.0 537.0 0.0 1.0 0.0 121.7 4517.07 4323.23 8162 0.553 0 0 L1 2/0 81.54 MB 0.3 1186.1 537.0 649.1 649.1 0.0 0.0 1.2 167.5 91.7 7251.22 5317.44 8161 0.889 9513M 5514M Sum 2/0 81.54 MB 0.0 1186.1 537.0 649.1 1186.1 537.0 0.0 2.2 103.2 103.2 11768.28 9640.68 16323 0.721 9513M 5514M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Updated by Igor Fedotov 7 months ago
hmm... I presume RocksDB numbers are inaccurate. Perhaps it doesn't take removed data records into account or something.
As a brute force investigation approach I'd suggest to export bluefs via ceph-bluestore-tool's bluefs-export command and learn the actual size it takes. Also you might try to compact the DB and check what would be the stats afterwards...
Updated by yite gu 7 months ago
Igor Fedotov wrote:
hmm... I presume RocksDB numbers are inaccurate. Perhaps it doesn't take removed data records into account or something.
As a brute force investigation approach I'd suggest to export bluefs via ceph-bluestore-tool's bluefs-export command and learn the actual size it takes. Also you might try to compact the DB and check what would be the stats afterwards...
1. I compacted all DB, the stats as below:
osd df:
$ ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 10 KiB 18 GiB 1.1 TiB 83.54 1.00 99 up
4 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 13 KiB 18 GiB 1.1 TiB 83.56 1.00 100 up
8 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 13 KiB 19 GiB 902 GiB 87.39 1.04 103 up
11 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 10 KiB 18 GiB 1.2 TiB 83.53 1.00 101 up
14 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 11 KiB 19 GiB 895 GiB 87.48 1.04 100 up
17 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 18 KiB 18 GiB 1.1 TiB 83.58 1.00 99 up
20 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 13 MiB 19 GiB 1.0 TiB 85.58 1.02 100 up
32 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 17 KiB 17 GiB 1.4 TiB 80.54 0.96 100 up
0 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 12 KiB 18 GiB 1.1 TiB 84.50 1.01 100 up
3 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 28 KiB 18 GiB 1.0 TiB 85.43 1.02 100 up
6 nvme 6.98630 1.00000 7.0 TiB 6.3 TiB 6.3 TiB 29 KiB 20 GiB 685 GiB 90.43 1.08 100 up
9 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 13 KiB 18 GiB 1.2 TiB 82.65 0.99 101 up
12 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 24 KiB 17 GiB 1.4 TiB 80.62 0.96 100 up
15 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 7 KiB 18 GiB 1.1 TiB 84.51 1.01 100 up
18 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 6 KiB 18 GiB 1.1 TiB 84.43 1.01 99 up
30 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 15 KiB 18 GiB 1.1 TiB 83.55 1.00 101 up
2 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 15 KiB 18 GiB 1.3 TiB 81.54 0.97 100 up
5 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 982 KiB 18 GiB 1.2 TiB 83.51 1.00 101 up
7 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 23 KiB 19 GiB 978 GiB 86.33 1.03 101 up
10 nvme 6.98630 1.00000 7.0 TiB 6.3 TiB 6.3 TiB 8 KiB 20 GiB 686 GiB 90.41 1.08 99 up
13 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 16 KiB 18 GiB 1.1 TiB 84.58 1.01 100 up
16 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 20 KiB 17 GiB 1.4 TiB 80.54 0.96 101 up
19 nvme 6.98630 1.00000 7.0 TiB 5.5 TiB 5.5 TiB 8 KiB 17 GiB 1.5 TiB 78.58 0.94 100 up
31 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 1000 KiB 18 GiB 1.3 TiB 81.61 0.97 100 up
23 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 15 KiB 18 GiB 1.2 TiB 83.49 1.00 100 up
35 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 14 KiB 18 GiB 1.2 TiB 83.52 1.00 99 up
44 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 986 KiB 18 GiB 1.2 TiB 83.46 1.00 100 up
53 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 14 KiB 18 GiB 1.1 TiB 84.41 1.01 101 up
62 nvme 6.98630 1.00000 7.0 TiB 5.2 TiB 5.2 TiB 1000 KiB 16 GiB 1.8 TiB 74.49 0.89 100 up
71 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 15 KiB 18 GiB 1.3 TiB 81.66 0.97 99 up
80 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 12 KiB 18 GiB 1.0 TiB 85.46 1.02 100 up
89 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 13 MiB 18 GiB 1.1 TiB 84.50 1.01 99 up
25 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 16 KiB 18 GiB 1.2 TiB 82.56 0.98 100 up
36 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 8 KiB 18 GiB 1.4 TiB 80.60 0.96 99 up
46 nvme 6.98630 1.00000 7.0 TiB 6.4 TiB 6.4 TiB 14 KiB 23 GiB 621 GiB 91.32 1.09 101 up
55 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 9 KiB 17 GiB 1.4 TiB 80.57 0.96 99 up
64 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 15 KiB 19 GiB 898 GiB 87.45 1.04 100 up
73 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 12 KiB 18 GiB 1.1 TiB 84.53 1.01 99 up
82 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 24 KiB 18 GiB 1.2 TiB 83.33 0.99 101 up
91 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 11 KiB 19 GiB 1.0 TiB 85.56 1.02 100 up
22 nvme 6.98630 1.00000 7.0 TiB 6.2 TiB 6.2 TiB 17 KiB 19 GiB 833 GiB 88.35 1.05 100 up
34 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 29 KiB 19 GiB 904 GiB 87.37 1.04 103 up
43 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 8 KiB 18 GiB 1.1 TiB 83.64 1.00 100 up
52 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 12 KiB 18 GiB 1.2 TiB 83.46 1.00 100 up
61 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 10 KiB 18 GiB 1.2 TiB 82.54 0.98 99 up
70 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 9 KiB 17 GiB 1.4 TiB 79.73 0.95 98 up
79 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 22 KiB 19 GiB 972 GiB 86.42 1.03 100 up
88 nvme 6.98630 1.00000 7.0 TiB 5.4 TiB 5.4 TiB 20 KiB 18 GiB 1.6 TiB 77.57 0.93 100 up
TOTAL 335 TiB 281 TiB 280 TiB 30 MiB 875 GiB 54 TiB 83.84
osd.88 perf dump:
"bluefs": {
"db_total_bytes": 7681501126656,
"db_used_bytes": 18912141312,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 0,
"slow_used_bytes": 0,
"num_files": 313,
"log_bytes": 4935680,
"log_compactions": 2,
"logged_bytes": 30945280,
"files_written_wal": 5,
"files_written_sst": 506,
"bytes_written_wal": 38068224,
"bytes_written_sst": 32528220160,
"bytes_written_slow": 0,
"max_bytes_wal": 0,
"max_bytes_db": 219581239296,
"max_bytes_slow": 0,
"read_random_count": 89446,
"read_random_bytes": 43336365348,
"read_random_disk_count": 70823,
"read_random_disk_bytes": 43104689672,
"read_random_buffer_count": 55903,
"read_random_buffer_bytes": 231675676,
"read_count": 25533,
"read_bytes": 1047650237,
"read_prefetch_count": 7694,
"read_prefetch_bytes": 479221816,
$ ceph tell osd.88 bluestore bluefs device info
{
"dev": {
"device": "BDEV_DB",
"total": 7681501126656,
"free": 1723064070144,
"bluefs_used": 18912141312,
"bluefs max available": 1723064070144
}
}
Everything seems normal now.
2. I run performance test now. I would bluefs-export after reproduct. It may take 1 day.
Updated by yite gu 7 months ago
osd.88 rocksdb stats now:
** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 58.4 0.12 0.00 2 0.062 0 0 L1 0/0 0.00 KB 0.0 0.5 0.1 0.5 0.5 0.0 0.0 7.3 57.1 53.6 9.45 8.35 2 4.724 16M 1009K L2 7/0 468.96 MB 0.2 0.7 0.2 0.5 0.5 -0.0 0.0 1.9 68.8 44.7 10.50 9.19 1 10.502 22M 7905K Sum 7/0 468.96 MB 0.0 1.2 0.3 0.9 1.0 0.0 0.0 136.3 62.9 49.0 20.07 17.54 5 4.015 39M 8915K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [m-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.5 0.00 0.00 1 0.001 0 0 L1 8/0 11.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 8/0 11.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.5 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 5/0 5.68 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 36.8 0.0 0.00 0.00 1 0.000 33 33 Sum 5/0 5.68 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 36.8 0.0 0.00 0.00 1 0.000 33 33 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [p-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L1 2/0 3.68 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 3.68 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 478.3 0.16 0.00 2 0.079 0 0 L1 0/0 0.00 KB 0.0 0.7 0.3 0.4 0.7 0.2 0.0 2.4 134.4 127.6 5.50 3.63 2 2.751 1084K 68K L2 0/0 0.00 KB 0.0 3.7 0.4 3.3 3.4 0.2 0.0 7.8 134.5 125.0 28.20 19.03 4 7.049 5109K 410K L3 77/0 4.92 GB 0.2 7.9 2.7 5.2 5.2 0.0 0.0 2.0 156.6 103.7 51.57 35.81 4 12.894 10M 3637K Sum 77/0 4.92 GB 0.0 12.3 3.4 8.9 9.4 0.5 0.0 127.3 147.6 112.9 85.43 58.47 12 7.119 17M 4116K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 555.7 0.14 0.00 2 0.068 0 0 L1 0/0 0.00 KB 0.0 0.7 0.3 0.4 0.7 0.2 0.0 2.3 127.5 119.7 5.63 3.66 2 2.814 1051K 81K L2 0/0 0.00 KB 0.0 4.2 0.5 3.7 3.9 0.2 0.0 8.4 146.3 136.4 29.25 20.06 6 4.876 5724K 439K L3 77/0 4.91 GB 0.2 7.8 2.7 5.2 5.2 0.0 0.0 1.9 154.9 102.0 51.90 35.92 4 12.975 10M 3634K Sum 77/0 4.91 GB 0.0 12.7 3.4 9.3 9.8 0.5 0.0 133.1 150.0 115.4 86.92 59.64 14 6.208 17M 4155K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 555.5 0.14 0.00 2 0.068 0 0 L1 0/0 0.00 KB 0.0 0.7 0.3 0.4 0.7 0.2 0.0 2.4 133.6 126.2 5.51 3.60 2 2.756 1076K 75K L2 0/0 0.00 KB 0.0 4.0 0.5 3.6 3.7 0.2 0.0 8.2 112.8 104.8 36.48 25.06 5 7.295 5490K 438K L3 77/0 4.92 GB 0.2 7.9 2.7 5.2 5.2 0.0 0.0 1.9 133.3 88.1 60.47 43.37 4 15.117 10M 3614K Sum 77/0 4.92 GB 0.0 12.6 3.4 9.2 9.7 0.5 0.0 132.1 125.8 96.7 102.59 72.03 13 7.892 17M 4129K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [L] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 2 0.002 0 0 L1 46/0 230.13 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 0.0 689.8 0.0 0.00 0.00 1 0.000 8 8 Sum 46/0 230.13 KB 0.0 0.0 0.0 0.0 0.0 -0.0 0.0 1.0 11.0 2.9 0.00 0.00 3 0.001 8 8 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [P] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 269.5 0.13 0.01 2 0.064 0 0 L1 2/0 81.45 MB 0.3 0.2 0.0 0.2 0.2 -0.0 0.0 4.7 373.1 307.4 0.53 0.52 2 0.265 1247K 267K Sum 2/0 81.45 MB 0.0 0.2 0.0 0.2 0.2 0.0 0.0 5.7 300.1 300.0 0.66 0.53 4 0.165 1247K 267K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Updated by yite gu 7 months ago
performance test is already running 33 hours. osd df as below:
[rook@n049-079-175-212 /]$ ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
1 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 15 KiB 27 GiB 1.1 TiB 83.68 1.00 99 up
4 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 16 KiB 27 GiB 1.1 TiB 83.69 1.00 100 up
8 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 25 KiB 28 GiB 893 GiB 87.52 1.04 103 up
11 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 15 KiB 28 GiB 1.1 TiB 83.66 1.00 101 up
14 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 19 KiB 30 GiB 884 GiB 87.64 1.04 100 up
17 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 18 KiB 28 GiB 1.1 TiB 83.72 1.00 99 up
20 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 13 MiB 31 GiB 1019 GiB 85.75 1.02 100 up
32 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 26 KiB 27 GiB 1.3 TiB 80.68 0.96 100 up
0 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 18 KiB 28 GiB 1.1 TiB 84.63 1.01 100 up
3 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 29 KiB 28 GiB 1.0 TiB 85.56 1.02 100 up
6 nvme 6.98630 1.00000 7.0 TiB 6.3 TiB 6.3 TiB 29 KiB 31 GiB 673 GiB 90.59 1.08 100 up
9 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 16 KiB 30 GiB 1.2 TiB 82.82 0.99 101 up
12 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 42 KiB 27 GiB 1.3 TiB 80.75 0.96 100 up
15 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 7 KiB 29 GiB 1.1 TiB 84.66 1.01 100 up
18 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 6 KiB 29 GiB 1.1 TiB 84.58 1.01 99 up
30 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 21 KiB 28 GiB 1.1 TiB 83.69 1.00 101 up
2 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 21 KiB 26 GiB 1.3 TiB 81.66 0.97 100 up
5 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 1.0 MiB 28 GiB 1.1 TiB 83.65 1.00 101 up
7 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 38 KiB 29 GiB 968 GiB 86.47 1.03 101 up
10 nvme 6.98630 1.00000 7.0 TiB 6.3 TiB 6.3 TiB 8 KiB 29 GiB 676 GiB 90.55 1.08 99 up
13 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 16 KiB 31 GiB 1.1 TiB 84.76 1.01 100 up
16 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 28 KiB 28 GiB 1.3 TiB 80.69 0.96 101 up
19 nvme 6.98630 1.00000 7.0 TiB 5.5 TiB 5.5 TiB 8 KiB 26 GiB 1.5 TiB 78.71 0.94 100 up
31 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 1.0 MiB 27 GiB 1.3 TiB 81.75 0.97 100 up
23 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 17 KiB 27 GiB 1.1 TiB 83.61 1.00 100 up
35 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 21 KiB 28 GiB 1.1 TiB 83.66 1.00 99 up
44 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 1.0 MiB 34 GiB 1.1 TiB 83.68 1.00 100 up
53 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 19 KiB 29 GiB 1.1 TiB 84.55 1.01 101 up
62 nvme 6.98630 1.00000 7.0 TiB 5.2 TiB 5.2 TiB 1.0 MiB 26 GiB 1.8 TiB 74.63 0.89 100 up
71 nvme 6.98630 1.00000 7.0 TiB 5.7 TiB 5.7 TiB 23 KiB 28 GiB 1.3 TiB 81.81 0.97 99 up
80 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 17 KiB 29 GiB 1.0 TiB 85.60 1.02 100 up
89 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 13 MiB 29 GiB 1.1 TiB 84.66 1.01 99 up
25 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 17 KiB 32 GiB 1.2 TiB 82.75 0.99 100 up
36 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 8 KiB 26 GiB 1.3 TiB 80.72 0.96 99 up
46 nvme 6.98630 1.00000 7.0 TiB 6.4 TiB 6.4 TiB 18 KiB 34 GiB 610 GiB 91.47 1.09 101 up
55 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 9 KiB 29 GiB 1.3 TiB 80.73 0.96 99 up
64 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 23 KiB 31 GiB 885 GiB 87.62 1.04 100 up
73 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.9 TiB 16 KiB 29 GiB 1.1 TiB 84.68 1.01 99 up
82 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 29 KiB 28 GiB 1.2 TiB 83.48 0.99 101 up
91 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 11 KiB 28 GiB 1023 GiB 85.70 1.02 100 up
22 nvme 6.98630 1.00000 7.0 TiB 6.2 TiB 6.2 TiB 27 KiB 29 GiB 823 GiB 88.49 1.05 100 up
34 nvme 6.98630 1.00000 7.0 TiB 6.1 TiB 6.1 TiB 36 KiB 28 GiB 895 GiB 87.49 1.04 103 up
43 nvme 6.98630 1.00000 7.0 TiB 5.9 TiB 5.8 TiB 8 KiB 30 GiB 1.1 TiB 83.81 1.00 100 up
52 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.8 TiB 20 KiB 29 GiB 1.1 TiB 83.61 1.00 100 up
61 nvme 6.98630 1.00000 7.0 TiB 5.8 TiB 5.7 TiB 12 KiB 28 GiB 1.2 TiB 82.68 0.98 99 up
70 nvme 6.98630 1.00000 7.0 TiB 5.6 TiB 5.6 TiB 12 KiB 27 GiB 1.4 TiB 79.86 0.95 98 up
79 nvme 6.98630 1.00000 7.0 TiB 6.0 TiB 6.0 TiB 24 KiB 28 GiB 962 GiB 86.55 1.03 100 up
88 nvme 6.98630 1.00000 7.0 TiB 5.4 TiB 5.4 TiB 28 KiB 27 GiB 1.6 TiB 77.70 0.93 100 up
TOTAL 335 TiB 282 TiB 280 TiB 31 MiB 1.3 TiB 54 TiB 83.99
MIN/MAX VAR: 0.89/1.09 STDDEV: 3.19
osd.88 perf dump
"bluefs": {
"db_total_bytes": 7681501126656,
"db_used_bytes": 28729425920,
"wal_total_bytes": 0,
"wal_used_bytes": 0,
"slow_total_bytes": 0,
"slow_used_bytes": 0,
"num_files": 442,
"log_bytes": 12300288,
"log_compactions": 88258,
"logged_bytes": 1462521335808,
"files_written_wal": 11944,
"files_written_sst": 109582,
"bytes_written_wal": 3913517834240,
"bytes_written_sst": 7575278858240,
"bytes_written_slow": 0,
"max_bytes_wal": 0,
"max_bytes_db": 219581239296,
"max_bytes_slow": 0,
"read_random_count": 908406827,
"read_random_bytes": 11193391440067,
"read_random_disk_count": 913483859,
"read_random_disk_bytes": 11187406700651,
"read_random_buffer_count": 478355,
"read_random_buffer_bytes": 5984739416,
"read_count": 134607,
"read_bytes": 48021127527,
"read_prefetch_count": 116768,
"read_prefetch_bytes": 47452699106,
$ ceph tell osd.88 bluestore bluefs device info
{
"dev": {
"device": "BDEV_DB",
"total": 7681501126656,
"free": 1713107165184,
"bluefs_used": 28858605568,
"bluefs max available": 1713107165184
}
}
There have been no osd of high db usage.
Updated by yite gu 7 months ago
now, I pick osd.20 to export bluefs.
1. db usage 31G from osd.20 meta stats
2. db usage from rocksdb stats:
** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 58.4 58.4 0.0 1.0 0.0 39.5 1515.69 1484.22 2536 0.598 0 0 L1 3/0 201.06 MB 0.8 194.5 58.4 136.0 158.2 22.2 0.0 2.7 37.3 30.4 5336.97 4584.73 634 8.418 6197M 1140M L2 11/0 468.98 MB 0.2 65.4 22.0 43.5 43.5 0.0 0.0 2.0 50.1 33.3 1337.17 1183.99 325 4.114 2092M 702M Sum 14/0 670.04 MB 0.0 259.9 80.4 179.5 260.1 80.6 0.0 4.5 32.5 32.5 8189.83 7252.94 3495 2.343 8289M 1842M Int 0/0 0.00 KB 0.0 0.8 0.2 0.6 0.8 0.2 0.0 5.5 35.8 34.2 23.43 20.48 10 2.343 26M 5652K ** Compaction Stats [O-0] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 197.62 MB 0.8 0.0 0.0 0.0 482.9 482.9 0.0 1.0 0.0 287.3 1721.14 1553.06 2675 0.643 0 0 L1 4/0 242.43 MB 0.9 756.5 482.7 273.8 620.0 346.2 0.0 1.3 202.2 165.7 3832.22 2843.17 1267 3.025 1078M 234M L2 47/0 1.95 GB 0.8 1737.7 345.6 1392.1 1393.7 1.6 0.4 4.0 148.4 119.0 11990.00 7653.32 5610 2.137 2302M 462M L3 86/0 5.45 GB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 138/0 7.83 GB 0.0 2494.3 828.3 1665.9 2496.6 830.7 0.4 5.2 145.6 145.7 17543.35 12049.54 9552 1.837 3381M 697M Int 0/0 0.00 KB 0.0 6.8 2.2 4.7 6.9 2.2 0.0 5.1 149.2 150.1 46.83 31.43 26 1.801 9226K 1881K ** Compaction Stats [O-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 196.04 MB 0.8 0.0 0.0 0.0 479.9 479.9 0.0 1.0 0.0 294.9 1666.34 1494.36 2536 0.657 0 0 L1 4/0 202.04 MB 0.8 746.0 479.8 266.3 612.6 346.3 0.0 1.3 204.4 167.9 3736.66 2868.35 1267 2.949 1067M 229M L2 44/0 1.94 GB 0.8 1742.3 345.7 1396.6 1398.1 1.6 0.4 4.0 146.4 117.5 12184.41 7701.77 5603 2.175 2313M 463M L3 85/0 5.45 GB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 134/0 7.78 GB 0.0 2488.3 825.5 1662.8 2490.6 827.8 0.4 5.2 144.9 145.0 17587.41 12064.49 9406 1.870 3381M 693M Int 0/0 0.00 KB 0.0 6.7 2.1 4.6 6.8 2.2 0.0 5.0 150.1 150.6 46.03 30.89 26 1.770 9120K 1895K ** Compaction Stats [O-2] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 197.13 MB 0.8 0.0 0.0 0.0 480.2 480.2 0.0 1.0 0.0 297.2 1654.84 1477.12 2536 0.653 0 0 L1 4/0 242.95 MB 0.9 752.3 480.0 272.2 616.6 344.4 0.0 1.3 199.6 163.6 3859.15 3015.89 1267 3.046 1072M 233M L2 45/0 1.94 GB 0.8 1730.9 343.7 1387.2 1388.7 1.6 0.4 4.0 147.9 118.7 11982.22 7587.17 5612 2.135 2295M 460M L3 85/0 5.44 GB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 135/0 7.81 GB 0.0 2483.1 823.8 1659.4 2485.5 826.2 0.4 5.2 145.3 145.5 17496.22 12080.18 9415 1.858 3368M 693M Int 0/0 0.00 KB 0.0 7.1 2.2 4.8 7.0 2.2 0.0 5.2 151.4 150.9 47.68 32.39 27 1.766 9523K 1960K
other column families usage very small, we could ignore.
670MB + 7.83 GB + 7.78 GB + 7.81 GB = 24.07GB
3. bluefs-export
# ceph-bluestore-tool bluefs-export --path /var/lib/ceph/osd/ceph-20 --out-dir ./osd-20-db-export # du -h ./osd-20-db-export/ 8.0K ./osd-20-db-export/sharding 4.0K ./osd-20-db-export/db.slow 25G ./osd-20-db-export/db 255M ./osd-20-db-export/db.wal 25G ./osd-20-db-export/
Actions