Actions
Bug #40306
closedPool dont show their true size after add more osd - Max Available 1TB
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hi
Last night we added 4 disk class ssd to our cluster in a host.
We added normaly without problem but at dashboard instanly we saw that pools VMS / VOLUMES / Images dont show their real size.
With the current pool size 2 will be near 5-6TB , an now just report 1TB.
Their current use and available space is incorrect calculated.
What we dont want to happen:
Hypervisors are still running the rbd disk in that pools, but they think and the cluster think that the're no space available as you see in the logs attached it show that pools only have 1 TB available.
What we done yet to try fix it:
- Reboot one by one the monitors.
- Reboot osd class SSD process.
Im really scared about ceph for some bug or something delete or discard the current rbd in usage.
OS: Centos 7.6
Linux CEPH-MON01 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29 14:59:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)
[root@CEPH-MON01 ~]# ceph -s
cluster:
id: e1ee8086-7cce-43fd-a252-3d677af22428
health: HEALTH_OK
services:
mon: 4 daemons, quorum CEPH001,CEPH002,CEPH003,CEPH-MON01 (age 2h)
mgr: CEPH001(active, since 26h)
osd: 92 osds: 92 up (since 22m), 92 in (since 36h)
rgw: 1 daemon active (ceph-rgw03)
data:
pools: 17 pools, 10872 pgs
objects: 114.43M objects, 292 TiB
usage: 351 TiB used, 74 TiB / 426 TiB avail
pgs: 10848 active+clean
21 active+clean+scrubbing+deep
3 active+clean+scrubbing
io:
client: 553 MiB/s rd, 37 MiB/s wr, 2.85k op/s rd, 294 op/s w
Ceph osd df tree
[root@CEPH-MON01 ~]# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-41 392.90399 - 393 TiB 330 TiB 329 TiB 49 MiB 605 GiB 63 TiB 0 0 - root archive
-37 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host CEPH-ARCH-R03-07
-23 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host CEPH005
-17 130.96800 - 131 TiB 110 TiB 109 TiB 13 MiB 202 GiB 21 TiB 83.64 1.01 - host CEPH006
7 archive 10.91399 1.00000 11 TiB 9.3 TiB 9.3 TiB 3.3 MiB 17 GiB 1.6 TiB 85.13 1.03 228 up osd.7
8 archive 10.91399 1.00000 11 TiB 8.7 TiB 8.7 TiB 3.2 MiB 15 GiB 2.2 TiB 80.03 0.97 228 up osd.8
9 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.1 TiB 5.1 MiB 18 GiB 1.8 TiB 83.89 1.02 228 up osd.9
10 archive 10.91399 1.00000 11 TiB 9.3 TiB 9.3 TiB 68 KiB 18 GiB 1.6 TiB 85.48 1.04 228 up osd.10
12 archive 10.91399 1.00000 11 TiB 8.7 TiB 8.7 TiB 16 KiB 17 GiB 2.2 TiB 79.64 0.96 226 up osd.12
13 archive 10.91399 1.00000 11 TiB 9.0 TiB 9.0 TiB 64 KiB 18 GiB 1.9 TiB 82.79 1.00 227 up osd.13
42 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.1 TiB 176 KiB 16 GiB 1.8 TiB 83.66 1.01 228 up osd.42
43 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.2 TiB 232 KiB 17 GiB 1.7 TiB 84.23 1.02 228 up osd.43
51 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.2 TiB 192 KiB 17 GiB 1.7 TiB 84.71 1.03 227 up osd.51
53 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.1 TiB 176 KiB 16 GiB 1.8 TiB 83.37 1.01 227 up osd.53
76 archive 10.91399 1.00000 11 TiB 9.6 TiB 9.5 TiB 120 KiB 18 GiB 1.4 TiB 87.57 1.06 227 up osd.76
80 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.1 TiB 144 KiB 16 GiB 1.8 TiB 83.18 1.01 229 up osd.80
-26 130.96800 - 131 TiB 109 TiB 109 TiB 13 MiB 200 GiB 22 TiB 83.54 1.01 - host CEPH007
14 archive 10.91399 1.00000 11 TiB 8.7 TiB 8.7 TiB 4 KiB 17 GiB 2.2 TiB 79.99 0.97 228 up osd.14
15 archive 10.91399 1.00000 11 TiB 8.8 TiB 8.8 TiB 132 KiB 16 GiB 2.1 TiB 80.75 0.98 227 up osd.15
16 archive 10.91399 1.00000 11 TiB 9.7 TiB 9.7 TiB 196 KiB 17 GiB 1.2 TiB 88.83 1.08 227 up osd.16
39 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.0 TiB 6.4 MiB 16 GiB 1.9 TiB 82.95 1.00 228 up osd.39
40 archive 10.91399 1.00000 11 TiB 9.5 TiB 9.5 TiB 46 KiB 17 GiB 1.4 TiB 87.37 1.06 227 up osd.40
44 archive 10.91399 1.00000 11 TiB 8.7 TiB 8.7 TiB 5.0 MiB 15 GiB 2.2 TiB 79.61 0.96 228 up osd.44
48 archive 10.91399 1.00000 11 TiB 9.5 TiB 9.5 TiB 192 KiB 17 GiB 1.4 TiB 87.38 1.06 227 up osd.48
49 archive 10.91399 1.00000 11 TiB 9.0 TiB 9.0 TiB 104 KiB 16 GiB 1.9 TiB 82.45 1.00 227 up osd.49
52 archive 10.91399 1.00000 11 TiB 9.5 TiB 9.5 TiB 4 KiB 19 GiB 1.4 TiB 86.99 1.05 227 up osd.52
77 archive 10.91399 1.00000 11 TiB 8.6 TiB 8.6 TiB 128 KiB 15 GiB 2.3 TiB 78.95 0.96 229 up osd.77
89 archive 10.91399 1.00000 11 TiB 9.0 TiB 9.0 TiB 232 KiB 16 GiB 1.9 TiB 82.34 1.00 228 up osd.89
90 archive 10.91399 1.00000 11 TiB 9.3 TiB 9.2 TiB 104 KiB 18 GiB 1.6 TiB 84.90 1.03 227 up osd.90
-31 130.96800 - 131 TiB 111 TiB 111 TiB 23 MiB 202 GiB 20 TiB 84.84 1.03 - host CEPH008
5 archive 10.91399 1.00000 11 TiB 9.4 TiB 9.4 TiB 5.4 MiB 17 GiB 1.5 TiB 85.95 1.04 227 up osd.5
6 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.1 TiB 5.7 MiB 16 GiB 1.8 TiB 83.20 1.01 228 up osd.6
11 archive 10.91399 1.00000 11 TiB 9.1 TiB 9.1 TiB 192 KiB 17 GiB 1.8 TiB 83.68 1.01 227 up osd.11
45 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.2 TiB 3.3 MiB 17 GiB 1.7 TiB 84.21 1.02 226 up osd.45
46 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.1 TiB 8.3 MiB 16 GiB 1.7 TiB 83.97 1.02 228 up osd.46
47 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.2 TiB 48 KiB 17 GiB 1.7 TiB 84.29 1.02 228 up osd.47
55 archive 10.91399 1.00000 11 TiB 9.3 TiB 9.2 TiB 128 KiB 18 GiB 1.7 TiB 84.84 1.03 228 up osd.55
70 archive 10.91399 1.00000 11 TiB 9.7 TiB 9.7 TiB 4 KiB 18 GiB 1.2 TiB 88.96 1.08 226 up osd.70
71 archive 10.91399 1.00000 11 TiB 8.9 TiB 8.9 TiB 196 KiB 16 GiB 2.0 TiB 81.53 0.99 229 up osd.71
78 archive 10.91399 1.00000 11 TiB 9.2 TiB 9.1 TiB 120 KiB 17 GiB 1.8 TiB 83.93 1.02 228 up osd.78
79 archive 10.91399 1.00000 11 TiB 9.6 TiB 9.6 TiB 72 KiB 17 GiB 1.3 TiB 88.08 1.07 228 up osd.79
91 archive 10.91399 1.00000 11 TiB 9.3 TiB 9.3 TiB 68 KiB 17 GiB 1.6 TiB 85.48 1.04 228 up osd.91
-1 28.04163 - 33 TiB 21 TiB 21 TiB 42 GiB 68 GiB 11 TiB 65.31 0.79 - root default
-30 6.23611 - 10 TiB 4.8 TiB 4.7 TiB 10 GiB 15 GiB 5.7 TiB 45.45 0.55 - host CEPH-SSD-004
4 ssd 0 1.00000 893 GiB 1.2 GiB 76 MiB 24 MiB 1000 MiB 892 GiB 0.13 0.00 0 up osd.4
41 ssd 0 1.00000 893 GiB 1.2 GiB 80 MiB 0 B 1 GiB 892 GiB 0.13 0.00 0 up osd.41
74 ssd 0.87209 1.00000 893 GiB 666 GiB 663 GiB 1.3 GiB 1.4 GiB 227 GiB 74.56 0.90 297 up osd.74
75 ssd 0.87209 1.00000 893 GiB 701 GiB 699 GiB 314 MiB 1.6 GiB 192 GiB 78.47 0.95 296 up osd.75
92 ssd 0.87299 1.00000 894 GiB 685 GiB 681 GiB 1.7 GiB 1.6 GiB 209 GiB 76.60 0.93 297 up osd.92
93 ssd 0.87299 1.00000 894 GiB 669 GiB 666 GiB 994 MiB 1.4 GiB 225 GiB 74.82 0.91 296 up osd.93
94 ssd 0.87299 1.00000 894 GiB 668 GiB 665 GiB 2.1 GiB 1.5 GiB 225 GiB 74.77 0.91 297 up osd.94
95 ssd 0.87299 1.00000 894 GiB 696 GiB 693 GiB 1.2 GiB 1.6 GiB 198 GiB 77.87 0.94 296 up osd.95
96 ssd 1.00000 1.00000 894 GiB 783 GiB 780 GiB 1.2 GiB 2.1 GiB 110 GiB 87.64 1.06 336 up osd.96
97 ssd 0 1.00000 894 GiB 1.2 GiB 88 MiB 303 MiB 721 MiB 893 GiB 0.13 0.00 0 up osd.97
98 ssd 0 1.00000 894 GiB 1.2 GiB 80 MiB 7.1 MiB 1017 MiB 893 GiB 0.13 0.00 0 up osd.98
99 ssd 0 1.00000 894 GiB 1.5 GiB 102 MiB 850 MiB 475 MiB 892 GiB 0.16 0.00 0 up osd.99
-3 6.97784 - 7.4 TiB 5.2 TiB 5.2 TiB 9.5 GiB 18 GiB 2.2 TiB 70.48 0.85 - host CEPH001
1 ssd 0.43599 1.00000 447 GiB 336 GiB 335 GiB 597 MiB 1.1 GiB 110 GiB 75.31 0.91 147 up osd.1
17 ssd 0.43599 1.00000 894 GiB 324 GiB 321 GiB 1.3 GiB 1.1 GiB 570 GiB 36.24 0.44 146 up osd.17
18 ssd 0.43599 1.00000 447 GiB 310 GiB 308 GiB 314 MiB 1.0 GiB 137 GiB 69.31 0.84 146 up osd.18
19 ssd 0.43599 1.00000 447 GiB 310 GiB 308 GiB 472 MiB 1.1 GiB 137 GiB 69.41 0.84 148 up osd.19
20 ssd 0.43599 1.00000 447 GiB 338 GiB 336 GiB 1.4 GiB 1.3 GiB 108 GiB 75.76 0.92 146 up osd.20
21 ssd 0.43599 1.00000 447 GiB 346 GiB 343 GiB 565 MiB 1.7 GiB 101 GiB 77.38 0.94 148 up osd.21
22 ssd 0.43599 1.00000 447 GiB 340 GiB 339 GiB 15 MiB 1.1 GiB 106 GiB 76.18 0.92 148 up osd.22
23 ssd 0.43599 1.00000 447 GiB 343 GiB 341 GiB 1011 MiB 1.4 GiB 104 GiB 76.78 0.93 147 up osd.23
37 ssd 0.87299 1.00000 894 GiB 671 GiB 668 GiB 1.1 GiB 1.7 GiB 223 GiB 75.04 0.91 293 up osd.37
54 ssd 0.87299 1.00000 894 GiB 682 GiB 679 GiB 1.0 GiB 1.6 GiB 212 GiB 76.28 0.92 294 up osd.54
56 ssd 0.43599 1.00000 447 GiB 330 GiB 329 GiB 447 MiB 1.1 GiB 116 GiB 73.96 0.90 146 up osd.56
60 ssd 0.43599 1.00000 447 GiB 350 GiB 348 GiB 175 KiB 1.1 GiB 97 GiB 78.26 0.95 147 up osd.60
61 ssd 0.43599 1.00000 447 GiB 353 GiB 351 GiB 217 MiB 1.3 GiB 94 GiB 78.95 0.96 148 up osd.61
62 ssd 0.43599 1.00000 447 GiB 321 GiB 319 GiB 1.1 GiB 1.1 GiB 126 GiB 71.82 0.87 146 up osd.62
-5 7.41383 - 7.4 TiB 5.7 TiB 5.6 TiB 11 GiB 18 GiB 1.8 TiB 76.33 0.92 - host CEPH002
2 ssd 0.43599 1.00000 447 GiB 363 GiB 362 GiB 924 MiB 1010 MiB 83 GiB 81.37 0.99 146 up osd.2
24 ssd 0.43599 1.00000 447 GiB 328 GiB 326 GiB 974 MiB 1.0 GiB 119 GiB 73.37 0.89 147 up osd.24
25 ssd 0.43599 1.00000 447 GiB 369 GiB 367 GiB 294 MiB 1.1 GiB 78 GiB 82.53 1.00 146 up osd.25
26 ssd 0.43599 1.00000 447 GiB 346 GiB 344 GiB 547 MiB 1003 MiB 101 GiB 77.45 0.94 148 up osd.26
27 ssd 0.43599 1.00000 447 GiB 321 GiB 319 GiB 1.5 GiB 1.2 GiB 125 GiB 71.93 0.87 146 up osd.27
28 ssd 0.43599 1.00000 447 GiB 372 GiB 371 GiB 6.2 MiB 1.0 GiB 74 GiB 83.32 1.01 147 up osd.28
29 ssd 0.43599 1.00000 447 GiB 342 GiB 341 GiB 39 MiB 1.0 GiB 104 GiB 76.61 0.93 149 up osd.29
30 ssd 0.43599 1.00000 447 GiB 315 GiB 314 GiB 235 MiB 927 MiB 131 GiB 70.64 0.86 147 up osd.30
38 ssd 0.87299 1.00000 894 GiB 687 GiB 684 GiB 1.1 GiB 1.7 GiB 206 GiB 76.90 0.93 292 up osd.38
57 ssd 0.43599 1.00000 447 GiB 326 GiB 324 GiB 827 MiB 1002 MiB 121 GiB 72.95 0.88 147 up osd.57
63 ssd 0.43599 1.00000 447 GiB 317 GiB 315 GiB 1.6 GiB 1.3 GiB 129 GiB 71.09 0.86 147 up osd.63
64 ssd 0.43599 1.00000 447 GiB 341 GiB 339 GiB 1.1 GiB 1.3 GiB 105 GiB 76.41 0.93 146 up osd.64
65 ssd 0.43599 1.00000 447 GiB 345 GiB 343 GiB 233 MiB 1.4 GiB 102 GiB 77.19 0.93 147 up osd.65
66 ssd 0.43599 1.00000 447 GiB 341 GiB 338 GiB 1.3 GiB 1.4 GiB 105 GiB 76.38 0.93 147 up osd.66
72 ssd 0.87299 1.00000 894 GiB 682 GiB 680 GiB 730 MiB 1.6 GiB 212 GiB 76.29 0.92 294 up osd.72
-7 7.41383 - 7.4 TiB 5.7 TiB 5.7 TiB 11 GiB 17 GiB 1.7 TiB 77.16 0.93 - host CEPH003
0 ssd 0.43599 1.00000 447 GiB 337 GiB 336 GiB 765 MiB 887 MiB 109 GiB 75.52 0.91 146 up osd.0
3 ssd 0.43599 1.00000 447 GiB 359 GiB 357 GiB 567 MiB 965 MiB 88 GiB 80.32 0.97 147 up osd.3
31 ssd 0.43599 1.00000 447 GiB 327 GiB 324 GiB 1.8 GiB 1.1 GiB 119 GiB 73.25 0.89 148 up osd.31
32 ssd 0.43599 1.00000 447 GiB 302 GiB 301 GiB 12 MiB 1012 MiB 144 GiB 67.72 0.82 147 up osd.32
33 ssd 0.43599 1.00000 447 GiB 354 GiB 353 GiB 32 MiB 999 MiB 92 GiB 79.32 0.96 145 up osd.33
34 ssd 0.43599 1.00000 447 GiB 365 GiB 364 GiB 316 MiB 973 MiB 82 GiB 81.73 0.99 146 up osd.34
35 ssd 0.43599 1.00000 447 GiB 341 GiB 338 GiB 1.7 GiB 891 MiB 106 GiB 76.27 0.92 147 up osd.35
36 ssd 0.43599 1.00000 447 GiB 368 GiB 366 GiB 865 MiB 1.2 GiB 79 GiB 82.31 1.00 148 up osd.36
50 ssd 0.87299 1.00000 894 GiB 680 GiB 675 GiB 2.6 GiB 1.8 GiB 214 GiB 76.07 0.92 292 up osd.50
58 ssd 0.43599 1.00000 447 GiB 344 GiB 343 GiB 182 MiB 998 MiB 102 GiB 77.06 0.93 147 up osd.58
59 ssd 0.43599 1.00000 447 GiB 356 GiB 355 GiB 59 MiB 965 MiB 90 GiB 79.75 0.97 148 up osd.59
67 ssd 0.43599 1.00000 447 GiB 360 GiB 358 GiB 412 MiB 1.4 GiB 87 GiB 80.58 0.98 148 up osd.67
68 ssd 0.43599 1.00000 447 GiB 323 GiB 321 GiB 826 MiB 1.3 GiB 124 GiB 72.30 0.88 145 up osd.68
69 ssd 0.43599 1.00000 447 GiB 349 GiB 347 GiB 962 MiB 986 MiB 98 GiB 78.11 0.95 147 up osd.69
73 ssd 0.87299 1.00000 894 GiB 694 GiB 692 GiB 411 MiB 1.6 GiB 199 GiB 77.69 0.94 294 up osd.73
TOTAL 426 TiB 351 TiB 351 TiB 42 GiB 673 GiB 74 TiB 82.57
MIN/MAX VAR: 0.00/1.08 STDDEV: 20.60
RAW STORAGE:
CLASS SIZE AVAIL USED RAW USED %RAW USED
archive 393 TiB 63 TiB 329 TiB 330 TiB 84.01
ssd 33 TiB 11 TiB 21 TiB 21 TiB 65.31
TOTAL 426 TiB 74 TiB 351 TiB 351 TiB 82.57
POOLS:
POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR
volumes 21 276 GiB 1.07M 554 GiB 20.90 1.0 TiB N/A N/A 1.07M 0 B 0 B
backups 22 0 B 0 0 B 0 1.0 TiB N/A N/A 0 0 B 0 B
images 23 147 GiB 291.71k 294 GiB 12.32 1.0 TiB N/A N/A 291.71k 0 B 0 B
vms 24 256 GiB 1.02M 513 GiB 19.67 1.0 TiB N/A N/A 1.02M 0 B 0 B
openstack-volumes-archive 25 46 TiB 12.07M 46 TiB 62.44 14 TiB N/A N/A 12.07M 0 B 0 B
.rgw.root 26 0 B 4 0 B 0 1.0 TiB N/A N/A 4 0 B 0 B
default.rgw.control 27 0 B 100 0 B 0 1.0 TiB N/A N/A 100 0 B 0 B
default.rgw.data.root 28 3.3 KiB 172 1.1 MiB 0 1.0 TiB N/A N/A 172 0 B 0 B
default.rgw.gc 29 559 MiB 32 559 MiB 0.03 1.0 TiB N/A N/A 32 0 B 0 B
default.rgw.log 30 7.3 KiB 175 7.3 KiB 0 1.0 TiB N/A N/A 175 0 B 0 B
default.rgw.users.uid 31 46 KiB 29 46 KiB 0 1.0 TiB N/A N/A 29 0 B 0 B
default.rgw.users.email 36 0 B 1 0 B 0 1.0 TiB N/A N/A 1 0 B 0 B
default.rgw.users.keys 37 21 B 16 192 KiB 0 1.0 TiB N/A N/A 16 0 B 0 B
default.rgw.buckets.index 38 29 GiB 1.20k 29 GiB 1.35 1.0 TiB N/A N/A 1.20k 0 B 0 B
default.rgw.buckets.data 39 235 TiB 99.98M 235 TiB 89.48 28 TiB N/A N/A 99.98M 0 B 0 B
default.rgw.buckets.non-ec 40 580 MiB 2.24k 582 MiB 0.03 1.0 TiB N/A N/A 2.24k 0 B 0 B
default.rgw.usage 43 6.0 MiB 8 6.0 MiB 0 1.0 TiB N/A N/A 8 0 B 0 B
Updated by Manuel Rios almost 5 years ago
We have reboot the SSD/monitors nodes, nothing change same pool size.
Updated by Manuel Rios almost 5 years ago
All SSD report this logs after run ceph-bluestore-tool repair
2019-06-17 02:33:29.483 7f924f419f00 -1 bluestore(/var/lib/ceph/osd/ceph-26) fsck error: missing Pool StatFS record for pool 18
2019-06-17 02:33:29.483 7f924f419f00 -1 bluestore(/var/lib/ceph/osd/ceph-26) fsck error: missing Pool StatFS record for pool 1c
2019-06-17 02:33:29.483 7f924f419f00 -1 bluestore(/var/lib/ceph/osd/ceph-26) fsck error: missing Pool StatFS record for pool ffffffffffffffff
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: legacy statfs record found, removing
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool 15
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool 17
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool 18
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool 1c
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool 25
2019-06-17 02:33:30.167 7fda9964ff00 -1 bluestore(/var/lib/ceph/osd/ceph-63) fsck error: missing Pool StatFS record for pool ffffffffffffffff
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: legacy statfs record found, removing
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: missing Pool StatFS record for pool 15
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: missing Pool StatFS record for pool 17
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: missing Pool StatFS record for pool 18
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: missing Pool StatFS record for pool 1c
2019-06-17 02:33:30.602 7f56b0344f00 -1 bluestore(/var/lib/ceph/osd/ceph-28) fsck error: missing Pool StatFS record for pool ffffffffffffffff
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: legacy statfs record found, removing
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool 15
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool 17
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool 18
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool 1c
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool 25
2019-06-17 02:33:34.503 7fa1f3c19f00 -1 bluestore(/var/lib/ceph/osd/ceph-65) fsck error: missing Pool StatFS record for pool ffffffffffffffff
This solve this bug, if someone got this error.
Stop your osd , run ceph-bluestore-tool repair --path /XXXX , start OSD and your stats will be fine.
Updated by Greg Farnum almost 5 years ago
- Project changed from Ceph to bluestore
- Category deleted (
OSD)
Updated by Igor Fedotov almost 5 years ago
- Status changed from New to 12
This is a known behavior in Nautilus. The mixture of new (created in Nautilus) and legacy OSDs results in such an inconsistent report due to changes in OSD statfs collection. OSD repair is currently the only way to fix this case.
Starting V14.2.2 there is a corresponding alert (see #39348).
So suggest to close with "Won't fix" label...
Updated by Thomas Kriechbaumer almost 5 years ago
Is there some kind of documentation in the docs, or release notes that would allow this to be called "known behaviour"? I only found here in the bugtracker and on a mailing-list thread...
Updated by Igor Fedotov almost 5 years ago
I'm not aware of any relevant documentation on the topic, may be it makes sense to fire a corresponding ticket...
Updated by Manuel Rios almost 5 years ago
Igor Fedotov wrote:
I'm not aware of any relevant documentation on the topic, may be it makes sense to fire a corresponding ticket...
Igor is a painful that anyone patched as 14.2.1.1 or put in the online documentation as Thomas said, a Know Behaviour as:
OSD created before Nautilus must run ceph-bluestore-tool repair --path in all devices if you experience missmatching data available after add a new OSD your CEPH.
As a ready product for companies know the real space used is vital to acomplish expenses.Even if ceph think that pool is full when its just a statistics failure.
I think the command should be added to the upgrade as all OSD must be restarted is a good why to solve in one touch that issue.
Regards
Manuel
Updated by Sage Weil over 4 years ago
- Status changed from 12 to Resolved
resolved by https://github.com/ceph/ceph/pull/28978, fixed in 14.2.2
Actions