Project

General

Profile

ceph_health_detail.log

ceph health details - Frank Li, 02/01/2018 04:28 AM

Download (13.3 KB)

 
1
HEALTH_WARN nodown flag(s) set; 32 osds down; 4 hosts (32 osds) down; 2 racks (32 osds) down; 1 row (32 osds) down; Reduced data availability: 10240 pgs inactive, 5 pgs down, 3431 pgs peering, 3643 pgs stale; Degraded data redundancy: 215798/552339 objects degraded (39.070%), 10240 pgs unclean, 4666 pgs degraded, 4666 pgs undersized
2
OSDMAP_FLAGS nodown flag(s) set
3
OSD_DOWN 32 osds down
4
    osd.0 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
5
    osd.1 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
6
    osd.2 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
7
    osd.3 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
8
    osd.4 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
9
    osd.5 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
10
    osd.6 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
11
    osd.7 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
12
    osd.8 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
13
    osd.9 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
14
    osd.10 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
15
    osd.11 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
16
    osd.12 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
17
    osd.13 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
18
    osd.14 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
19
    osd.15 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
20
    osd.16 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
21
    osd.17 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
22
    osd.18 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
23
    osd.19 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
24
    osd.20 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
25
    osd.21 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
26
    osd.22 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
27
    osd.23 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
28
    osd.24 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
29
    osd.25 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
30
    osd.26 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
31
    osd.27 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
32
    osd.28 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
33
    osd.29 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
34
    osd.30 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
35
    osd.31 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
36
OSD_HOST_DOWN 4 hosts (32 osds) down
37
    host dl1-cephosd101 (root=openstack,row=1,rack=602) (8 osds) is down
38
    host dl1-cephosd102 (root=openstack,row=1,rack=602) (8 osds) is down
39
    host dl1-cephosd201 (root=openstack,row=1,rack=603) (8 osds) is down
40
    host dl1-cephosd202 (root=openstack,row=1,rack=603) (8 osds) is down
41
OSD_RACK_DOWN 2 racks (32 osds) down
42
    rack 602 (root=openstack,row=1) (16 osds) is down
43
    rack 603 (root=openstack,row=1) (16 osds) is down
44
OSD_ROW_DOWN 1 row (32 osds) down
45
    row 1 (root=openstack) (32 osds) is down
46
PG_AVAILABILITY Reduced data availability: 10240 pgs inactive, 5 pgs down, 3431 pgs peering, 3643 pgs stale
47
    pg 1.1fcd is stuck inactive for 6415.071555, current state peering, last acting [10,28,55]
48
    pg 1.1fce is stuck inactive for 30221.951178, current state unknown, last acting []
49
    pg 1.1fcf is stuck inactive for 30221.951178, current state unknown, last acting []
50
    pg 1.1fd0 is stuck peering for 25786.183809, current state peering, last acting [50,59]
51
    pg 1.1fd1 is stuck stale for 11565.240195, current state stale+undersized+degraded+peered, last acting [6]
52
    pg 1.1fd2 is stuck peering for 25772.727420, current state peering, last acting [1,20,63]
53
    pg 1.1fd3 is stuck inactive for 30221.951178, current state unknown, last acting []
54
    pg 1.1fd4 is stuck stale for 11565.240197, current state stale+undersized+degraded+peered, last acting [29]
55
    pg 1.1fd5 is stuck stale for 11565.240197, current state stale+undersized+degraded+peered, last acting [7]
56
    pg 1.1fd6 is stuck stale for 11565.240198, current state stale+undersized+degraded+peered, last acting [9]
57
    pg 1.1fd7 is stuck stale for 11565.240198, current state stale+undersized+degraded+peered, last acting [2]
58
    pg 1.1fd8 is stuck stale for 11565.240199, current state stale+undersized+degraded+peered, last acting [9]
59
    pg 1.1fd9 is stuck stale for 11565.240200, current state stale+undersized+degraded+peered, last acting [3]
60
    pg 1.1fda is stuck stale for 11565.240200, current state stale+undersized+degraded+peered, last acting [31]
61
    pg 1.1fdb is stuck stale for 11565.240201, current state stale+undersized+degraded+peered, last acting [16]
62
    pg 1.1fdc is stuck inactive for 30221.951178, current state unknown, last acting []
63
    pg 1.1fdd is stuck peering for 25789.507905, current state peering, last acting [40,30]
64
    pg 1.1fde is stuck peering for 25783.103856, current state peering, last acting [51,63]
65
    pg 1.1fdf is stuck inactive for 25771.709997, current state undersized+degraded+peered, last acting [63]
66
    pg 1.1fe0 is stuck stale for 11565.240202, current state stale+undersized+degraded+peered, last acting [3]
67
    pg 1.1fe1 is stuck peering for 26523.274018, current state peering, last acting [40,37,11]
68
    pg 1.1fe2 is stuck stale for 7148.035823, current state stale+peering, last acting [13,63]
69
    pg 1.1fe3 is stuck peering for 25783.098959, current state peering, last acting [16,25]
70
    pg 1.1fe4 is stuck stale for 11565.240209, current state stale+undersized+degraded+peered, last acting [33]
71
    pg 1.1fe5 is stuck peering for 25786.195140, current state peering, last acting [47,6]
72
    pg 1.1fe6 is stuck inactive for 30221.951178, current state unknown, last acting []
73
    pg 1.1fe7 is stuck inactive for 30221.951178, current state unknown, last acting []
74
    pg 1.1fe8 is stuck stale for 11565.240210, current state stale+undersized+degraded+peered, last acting [29]
75
    pg 1.1fe9 is stuck stale for 11565.240211, current state stale+undersized+degraded+peered, last acting [1]
76
    pg 1.1fea is stuck peering for 25786.199662, current state peering, last acting [52,7]
77
    pg 1.1feb is stuck inactive for 30221.951178, current state unknown, last acting []
78
    pg 1.1fec is stuck stale for 11565.240211, current state stale+undersized+degraded+peered, last acting [29]
79
    pg 1.1fed is stuck inactive for 30221.951178, current state unknown, last acting []
80
    pg 1.1fee is stuck inactive for 30221.951178, current state unknown, last acting []
81
    pg 1.1fef is stuck inactive for 30221.951178, current state unknown, last acting []
82
    pg 1.1ff0 is stuck stale for 11565.240227, current state stale+undersized+degraded+peered, last acting [3]
83
    pg 1.1ff1 is stuck stale for 11565.240227, current state stale+undersized+degraded+peered, last acting [0]
84
    pg 1.1ff2 is stuck peering for 25770.632924, current state peering, last acting [9,3]
85
    pg 1.1ff3 is stuck inactive for 25782.074274, current state undersized+degraded+peered, last acting [57]
86
    pg 1.1ff4 is stuck peering for 25783.096624, current state peering, last acting [25,4]
87
    pg 1.1ff5 is stuck peering for 25776.467718, current state peering, last acting [52,31]
88
    pg 1.1ff6 is stuck stale for 11565.240237, current state stale+undersized+degraded+peered, last acting [25]
89
    pg 1.1ff7 is stuck peering for 25778.778252, current state peering, last acting [6,17]
90
    pg 1.1ff8 is stuck inactive for 30221.951178, current state unknown, last acting []
91
    pg 1.1ff9 is stuck stale for 11565.240257, current state stale+undersized+degraded+peered, last acting [6]
92
    pg 1.1ffa is stuck stale for 7148.035831, current state stale+peering, last acting [13,4]
93
    pg 1.1ffb is stuck peering for 25786.171740, current state peering, last acting [29,51]
94
    pg 1.1ffc is stuck inactive for 30221.951178, current state unknown, last acting []
95
    pg 1.1ffd is stuck inactive for 30221.951178, current state unknown, last acting []
96
    pg 1.1ffe is stuck stale for 11565.240263, current state stale+undersized+degraded+peered, last acting [9]
97
    pg 1.1fff is stuck inactive for 30221.951178, current state unknown, last acting []
98
PG_DEGRADED Degraded data redundancy: 215798/552339 objects degraded (39.070%), 10240 pgs unclean, 4666 pgs degraded, 4666 pgs undersized
99
    pg 1.1fcd is stuck unclean for 26213.638775, current state peering, last acting [10,28,55]
100
    pg 1.1fce is stuck unclean for 30221.951178, current state unknown, last acting []
101
    pg 1.1fcf is stuck unclean for 30221.951178, current state unknown, last acting []
102
    pg 1.1fd0 is stuck unclean for 25788.313231, current state peering, last acting [50,59]
103
    pg 1.1fd1 is stuck undersized for 15556.585572, current state stale+undersized+degraded+peered, last acting [6]
104
    pg 1.1fd2 is stuck unclean for 25772.727420, current state peering, last acting [1,20,63]
105
    pg 1.1fd3 is stuck unclean for 30221.951178, current state unknown, last acting []
106
    pg 1.1fd4 is stuck undersized for 15526.837130, current state stale+undersized+degraded+peered, last acting [29]
107
    pg 1.1fd5 is stuck undersized for 15556.288593, current state stale+undersized+degraded+peered, last acting [7]
108
    pg 1.1fd6 is stuck undersized for 15546.524192, current state stale+undersized+degraded+peered, last acting [9]
109
    pg 1.1fd7 is stuck undersized for 15556.592113, current state stale+undersized+degraded+peered, last acting [2]
110
    pg 1.1fd8 is stuck undersized for 15546.516976, current state stale+undersized+degraded+peered, last acting [9]
111
    pg 1.1fd9 is stuck undersized for 15556.827903, current state stale+undersized+degraded+peered, last acting [3]
112
    pg 1.1fda is stuck undersized for 15540.223467, current state stale+undersized+degraded+peered, last acting [31]
113
    pg 1.1fdb is stuck undersized for 15546.313751, current state stale+undersized+degraded+peered, last acting [16]
114
    pg 1.1fdc is stuck unclean for 30221.951178, current state unknown, last acting []
115
    pg 1.1fdd is stuck unclean for 26203.148219, current state peering, last acting [40,30]
116
    pg 1.1fde is stuck unclean for 25785.208621, current state peering, last acting [51,63]
117
    pg 1.1fdf is stuck undersized for 15546.647448, current state undersized+degraded+peered, last acting [63]
118
    pg 1.1fe0 is stuck undersized for 15556.836267, current state stale+undersized+degraded+peered, last acting [3]
119
    pg 1.1fe1 is stuck unclean for 26523.274018, current state peering, last acting [40,37,11]
120
    pg 1.1fe2 is stuck unclean for 25786.220114, current state stale+peering, last acting [13,63]
121
    pg 1.1fe3 is stuck unclean for 30097.504822, current state peering, last acting [16,25]
122
    pg 1.1fe4 is stuck undersized for 15546.482837, current state stale+undersized+degraded+peered, last acting [33]
123
    pg 1.1fe5 is stuck unclean for 30220.857111, current state peering, last acting [47,6]
124
    pg 1.1fe6 is stuck unclean for 30221.951178, current state unknown, last acting []
125
    pg 1.1fe7 is stuck unclean for 30221.951178, current state unknown, last acting []
126
    pg 1.1fe8 is stuck undersized for 15526.792522, current state stale+undersized+degraded+peered, last acting [29]
127
    pg 1.1fe9 is stuck undersized for 15556.007507, current state stale+undersized+degraded+peered, last acting [1]
128
    pg 1.1fea is stuck unclean for 30221.917471, current state peering, last acting [52,7]
129
    pg 1.1feb is stuck unclean for 30221.951178, current state unknown, last acting []
130
    pg 1.1fec is stuck undersized for 15526.686958, current state stale+undersized+degraded+peered, last acting [29]
131
    pg 1.1fed is stuck unclean for 30221.951178, current state unknown, last acting []
132
    pg 1.1fee is stuck unclean for 30221.951178, current state unknown, last acting []
133
    pg 1.1fef is stuck unclean for 30221.951178, current state unknown, last acting []
134
    pg 1.1ff0 is stuck undersized for 15556.819248, current state stale+undersized+degraded+peered, last acting [3]
135
    pg 1.1ff1 is stuck undersized for 15556.151991, current state stale+undersized+degraded+peered, last acting [0]
136
    pg 1.1ff2 is stuck unclean for 25772.739719, current state peering, last acting [9,3]
137
    pg 1.1ff3 is stuck undersized for 15546.926259, current state undersized+degraded+peered, last acting [57]
138
    pg 1.1ff4 is stuck unclean for 88019.620739, current state peering, last acting [25,4]
139
    pg 1.1ff5 is stuck unclean for 25778.781465, current state peering, last acting [52,31]
140
    pg 1.1ff6 is stuck undersized for 15540.377043, current state stale+undersized+degraded+peered, last acting [25]
141
    pg 1.1ff7 is stuck unclean for 25779.890723, current state peering, last acting [6,17]
142
    pg 1.1ff8 is stuck unclean for 30221.951178, current state unknown, last acting []
143
    pg 1.1ff9 is stuck undersized for 15556.575201, current state stale+undersized+degraded+peered, last acting [6]
144
    pg 1.1ffa is stuck unclean for 30199.625515, current state stale+peering, last acting [13,4]
145
    pg 1.1ffb is stuck unclean for 44171.550888, current state peering, last acting [29,51]
146
    pg 1.1ffc is stuck unclean for 30221.951178, current state unknown, last acting []
147
    pg 1.1ffd is stuck unclean for 30221.951178, current state unknown, last acting []
148
    pg 1.1ffe is stuck undersized for 15546.458395, current state stale+undersized+degraded+peered, last acting [9]
149
    pg 1.1fff is stuck unclean for 30221.951178, current state unknown, last acting []