Project

General

Profile

ceph-12.0-2.log

log of ceph -w and the output of ceph -s and ceph osd tree following execution of rados bench - Nathan Cutler, 03/03/2017 12:32 PM

Download (26.1 KB)

 
1
admin:~ # ceph -w
2
    cluster 557c4525-cad3-371a-8e05-4b95f6accdf5
3
     health HEALTH_OK
4
     monmap e2: 3 mons at {data1=172.16.1.21:6789/0,data2=172.16.1.22:6789/0,mon1=172.16.1.11:6789/0}
5
            election epoch 6, quorum 0,1,2 mon1,data1,data2
6
        mgr active: mon1 standbys: data1, data2
7
     osdmap e40: 8 osds: 8 up, 8 in
8
            flags sortbitwise,require_jewel_osds,require_kraken_osds,require_luminous_osds
9
      pgmap v96: 248 pgs, 1 pools, 0 bytes data, 0 objects
10
            282 MB used, 10407 MB / 10690 MB avail
11
                 248 active+clean
12

    
13
2017-03-02 16:00:00.000236 mon.0 [INF] HEALTH_OK
14
2017-03-02 16:00:16.868943 mon.0 [INF] pgmap v97: 248 pgs: 248 active+clean; 32768 kB data, 282 MB use
15
d, 10407 MB / 10690 MB avail; 1243 kB/s wr, 0 op/s
16
2017-03-02 16:00:25.485054 mon.0 [INF] pgmap v98: 248 pgs: 248 active+clean; 196 MB data, 1838 MB used
17
, 8852 MB / 10690 MB avail; 5992 kB/s wr, 2 op/s
18
2017-03-02 16:00:34.127780 mon.0 [INF] pgmap v99: 248 pgs: 248 active+clean; 384 MB data, 3349 MB used
19
, 7340 MB / 10690 MB avail; 9448 kB/s wr, 4 op/s
20
2017-03-02 16:00:45.216840 mon.0 [INF] pgmap v100: 248 pgs: 248 active+clean; 664 MB data, 4801 MB use
21
d, 5888 MB / 10690 MB avail; 13263 kB/s wr, 6 op/s
22
2017-03-02 16:00:42.398014 osd.1 [WRN] OSD near full (94%)
23
2017-03-02 16:00:54.949233 mon.0 [INF] pgmap v101: 248 pgs: 248 active+clean; 912 MB data, 6045 MB use
24
d, 4645 MB / 10690 MB avail; 15103 kB/s wr, 7 op/s
25
2017-03-02 16:00:51.798991 osd.1 [ERR] OSD full dropping all updates 100% full
26
2017-03-02 16:00:52.199787 osd.0 [WRN] OSD near full (91%)
27
2017-03-02 16:00:57.699552 osd.1 [WRN] OSD near full (97%)
28
2017-03-02 16:01:00.309164 mon.0 [INF] pgmap v102: 248 pgs: 248 active+clean; 1272 MB data, 7854 MB us
29
ed, 2836 MB / 10690 MB avail; 18507 kB/s wr, 9 op/s
30
2017-03-02 16:00:57.535252 osd.6 [WRN] OSD near full (96%)
31
2017-03-02 16:00:59.801120 osd.0 [ERR] OSD full dropping all updates 97% full
32
2017-03-02 16:01:00.201562 osd.3 [ERR] OSD full dropping all updates 100% full
33
2017-03-02 16:01:00.702151 osd.3 [WRN] OSD near full (93%)
34
2017-03-02 16:01:01.800024 osd.1 [ERR] OSD full dropping all updates 100% full
35
2017-03-02 16:01:02.481969 osd.5 [WRN] OSD near full (93%)
36
2017-03-02 16:01:03.901671 osd.0 [WRN] OSD near full (92%)
37
2017-03-02 16:01:04.202755 osd.3 [ERR] OSD full dropping all updates 100% full
38
2017-03-02 16:01:05.081565 osd.7 [WRN] OSD near full (91%)
39
2017-03-02 16:01:06.644936 mon.0 [INF] HEALTH_WARN; Difference in osd space utilization 42.3649% great
40
er than 40%; 1 near full osd(s)
41
2017-03-02 16:01:07.683188 osd.5 [ERR] OSD full dropping all updates 100% full
42
2017-03-02 16:01:09.455542 mon.0 [INF] pgmap v103: 248 pgs: 248 active+clean; 1440 MB data, 8403 MB us
43
ed, 2286 MB / 10690 MB avail; 28918 kB/s wr, 14 op/s
44
2017-03-02 16:01:09.455717 mon.0 [INF] osdmap e41: 8 osds: 8 up, 8 in nearfull
45
2017-03-02 16:01:09.682783 osd.7 [WRN] OSD near full (90%)
46
2017-03-02 16:01:10.291758 mon.0 [INF] pgmap v104: 248 pgs: 248 active+clean; 1668 MB data, 9007 MB us
47
ed, 1682 MB / 10690 MB avail; 30373 kB/s wr, 14 op/s
48
2017-03-02 16:01:11.379502 mon.0 [INF] pgmap v105: 248 pgs: 248 active+clean; 1816 MB data, 9447 MB us
49
ed, 1242 MB / 10690 MB avail; 33249 kB/s wr, 16 op/s
50
2017-03-02 16:01:13.279581 mon.0 [INF] osdmap e42: 8 osds: 8 up, 8 in full
51
2017-03-02 16:01:13.543353 mon.0 [INF] pgmap v106: 248 pgs: 248 active+clean; 1852 MB data, 9508 MB us
52
ed, 1182 MB / 10690 MB avail; 33421 kB/s wr, 16 op/s
53
2017-03-02 16:01:15.442087 mon.0 [INF] pgmap v107: 248 pgs: 248 active+clean; 1856 MB data, 9496 MB us
54
ed, 1193 MB / 10690 MB avail; 34650 kB/s wr, 16 op/s
55
2017-03-02 16:01:08.602416 osd.0 [ERR] OSD full dropping all updates 100% full
56
2017-03-02 16:01:16.514533 mon.0 [INF] pgmap v108: 248 pgs: 248 active+clean; 1856 MB data, 9380 MB us
57
ed, 1310 MB / 10690 MB avail; 29180 kB/s wr, 14 op/s
58
2017-03-02 16:01:15.304448 osd.3 [WRN] OSD near full (96%)
59
2017-03-02 16:01:17.584997 osd.5 [WRN] OSD near full (95%)
60
2017-03-02 16:01:18.771202 mon.0 [INF] pgmap v109: 248 pgs: 248 active+clean; 1856 MB data, 9229 MB us
61
ed, 1461 MB / 10690 MB avail; 25194 kB/s wr, 12 op/s
62
2017-03-02 16:01:19.842066 mon.0 [INF] pgmap v110: 248 pgs: 248 active+clean; 1856 MB data, 9149 MB us
63
ed, 1541 MB / 10690 MB avail; 17371 kB/s wr, 8 op/s
64
2017-03-02 16:01:20.922190 mon.0 [INF] pgmap v111: 248 pgs: 248 active+clean; 1856 MB data, 8939 MB us
65
ed, 1751 MB / 10690 MB avail; 4287 kB/s wr, 2 op/s
66
2017-03-02 16:01:23.015742 mon.0 [INF] pgmap v112: 248 pgs: 248 active+clean; 1856 MB data, 8938 MB us
67
ed, 1751 MB / 10690 MB avail; 422 kB/s wr, 0 op/s
68
2017-03-02 16:01:25.167305 mon.0 [INF] pgmap v113: 248 pgs: 248 active+clean; 1856 MB data, 8938 MB us
69
ed, 1751 MB / 10690 MB avail
70
2017-03-02 16:01:17.908801 osd.0 [WRN] OSD near full (95%)
71
2017-03-02 16:01:32.606021 osd.1 [ERR] OSD full dropping all updates 97% full
72
2017-03-02 16:01:47.711172 osd.3 [WRN] OSD near full (96%)
73
2017-03-02 16:01:48.489545 osd.5 [WRN] OSD near full (95%)
74
2017-03-02 16:01:51.913736 osd.0 [WRN] OSD near full (95%)
75
2017-03-02 16:02:06.645301 mon.0 [INF] HEALTH_ERR; 4 full osd(s); 1 near full osd(s); full flag(s) set
76
2017-03-02 16:02:03.313213 osd.1 [ERR] OSD full dropping all updates 97% full
77
2017-03-02 16:02:18.318061 osd.3 [WRN] OSD near full (96%)
78
2017-03-02 16:02:18.393507 osd.5 [WRN] OSD near full (95%)
79
2017-03-02 16:02:24.618909 osd.0 [WRN] OSD near full (95%)
80
2017-03-02 16:02:34.718579 osd.1 [ERR] OSD full dropping all updates 97% full
81
2017-03-02 16:02:51.698604 osd.5 [WRN] OSD near full (95%)
82
2017-03-02 16:02:53.621977 osd.3 [WRN] OSD near full (96%)
83
2017-03-02 16:02:57.025961 osd.0 [WRN] OSD near full (95%)
84
2017-03-02 16:03:08.023640 osd.1 [ERR] OSD full dropping all updates 97% full
85
2017-03-02 16:03:21.603128 osd.5 [WRN] OSD near full (95%)
86
2017-03-02 16:03:24.731983 osd.3 [WRN] OSD near full (96%)
87
2017-03-02 16:03:35.707623 mon.0 [INF] pgmap v114: 248 pgs: 248 active+clean; 1856 MB data, 8468 MB us
88
ed, 2221 MB / 10690 MB avail
89
2017-03-02 16:03:31.733471 osd.0 [WRN] OSD near full (95%)
90
2017-03-02 16:03:38.285830 mon.0 [INF] osdmap e43: 8 osds: 8 up, 8 in full
91
2017-03-02 16:03:38.361267 mon.0 [INF] pgmap v115: 248 pgs: 248 active+clean; 1856 MB data, 8468 MB us
92
ed, 2221 MB / 10690 MB avail
93
2017-03-02 16:03:40.368272 mon.0 [INF] pgmap v116: 248 pgs: 248 active+clean; 1856 MB data, 8445 MB us
94
ed, 2244 MB / 10690 MB avail
95
2017-03-02 16:03:41.444201 mon.0 [INF] pgmap v117: 248 pgs: 248 active+clean; 1856 MB data, 8437 MB us
96
ed, 2252 MB / 10690 MB avail
97
2017-03-02 16:03:43.300288 mon.0 [INF] osdmap e44: 8 osds: 8 up, 8 in full
98
2017-03-02 16:03:43.379195 mon.0 [INF] pgmap v118: 248 pgs: 248 active+clean; 1856 MB data, 8433 MB us
99
ed, 2256 MB / 10690 MB avail
100
2017-03-02 16:03:45.375373 mon.0 [INF] pgmap v119: 248 pgs: 248 active+clean; 1856 MB data, 8414 MB us
101
ed, 2276 MB / 10690 MB avail
102
2017-03-02 16:03:46.438494 mon.0 [INF] pgmap v120: 248 pgs: 248 active+clean; 1856 MB data, 8414 MB us
103
ed, 2276 MB / 10690 MB avail
104
2017-03-02 16:03:39.727425 osd.1 [ERR] OSD full dropping all updates 97% full
105
2017-03-02 16:03:45.027991 osd.1 [WRN] OSD near full (97%)
106
2017-03-02 16:03:48.644909 mon.0 [INF] pgmap v121: 248 pgs: 248 active+clean; 1856 MB data, 8410 MB us
107
ed, 2280 MB / 10690 MB avail
108
2017-03-02 16:03:50.753073 mon.0 [INF] pgmap v122: 248 pgs: 248 active+clean; 1856 MB data, 8382 MB us
109
ed, 2307 MB / 10690 MB avail
110
2017-03-02 16:03:53.028311 mon.0 [INF] pgmap v123: 248 pgs: 248 active+clean; 1856 MB data, 7944 MB us
111
ed, 2745 MB / 10690 MB avail
112
2017-03-02 16:03:55.129215 mon.0 [INF] pgmap v124: 248 pgs: 248 active+clean; 1856 MB data, 7944 MB us
113
ed, 2745 MB / 10690 MB avail
114
2017-03-02 16:03:51.411028 osd.5 [WRN] OSD near full (94%)
115
2017-03-02 16:03:54.338277 osd.3 [WRN] OSD near full (96%)
116
2017-03-02 16:04:06.645956 mon.0 [INF] HEALTH_ERR; Difference in osd space utilization 45.537% greater
117
 than 40%; 2 full osd(s); 2 near full osd(s); full flag(s) set
118
2017-03-02 16:04:01.339401 osd.0 [WRN] OSD near full (94%)
119
2017-03-02 16:04:15.234013 osd.1 [WRN] OSD near full (97%)
120
2017-03-02 16:04:22.416117 osd.5 [WRN] OSD near full (94%)
121
2017-03-02 16:04:25.342770 osd.3 [WRN] OSD near full (96%)
122
2017-03-02 16:04:33.846091 osd.0 [WRN] OSD near full (94%)
123
2017-03-02 16:04:47.839188 osd.1 [WRN] OSD near full (97%)
124
2017-03-02 16:04:52.721065 osd.5 [WRN] OSD near full (94%)
125
2017-03-02 16:04:55.246440 osd.3 [WRN] OSD near full (96%)
126
2017-03-02 16:05:04.151031 osd.0 [WRN] OSD near full (94%)
127
2017-03-02 16:05:19.983481 mon.0 [INF] pgmap v125: 248 pgs: 248 active+clean; 1856 MB data, 7642 MB us
128
ed, 3047 MB / 10690 MB avail
129
2017-03-02 16:05:21.143362 mon.0 [INF] pgmap v126: 248 pgs: 248 active+clean; 1856 MB data, 7300 MB us
130
ed, 3389 MB / 10690 MB avail
131
2017-03-02 16:05:25.625063 mon.0 [INF] pgmap v127: 248 pgs: 248 active+clean; 1856 MB data, 6863 MB us
132
ed, 3826 MB / 10690 MB avail
133
2017-03-02 16:05:24.825684 osd.5 [WRN] OSD near full (94%)
134
2017-03-02 16:05:28.352873 osd.3 [WRN] OSD near full (96%)
135
2017-03-02 16:05:30.286750 mon.0 [INF] pgmap v128: 248 pgs: 248 active+clean; 1856 MB data, 6539 MB us
136
ed, 4151 MB / 10690 MB avail
137
2017-03-02 16:05:33.326578 mon.0 [INF] osdmap e45: 8 osds: 8 up, 8 in nearfull
138
2017-03-02 16:05:33.767557 mon.0 [INF] pgmap v129: 248 pgs: 248 active+clean; 1856 MB data, 6539 MB us
139
ed, 4151 MB / 10690 MB avail
140
2017-03-02 16:05:44.907158 mon.0 [INF] pgmap v130: 248 pgs: 248 active+clean; 1944 MB data, 6119 MB us
141
ed, 4571 MB / 10690 MB avail; 895 kB/s wr, 0 op/s
142
2017-03-02 16:05:39.829279 osd.5 [WRN] OSD near full (96%)
143
2017-03-02 16:05:40.355809 osd.3 [WRN] OSD near full (96%)
144
2017-03-02 16:05:41.457745 osd.0 [WRN] OSD near full (95%)
145
2017-03-02 16:05:45.056642 osd.3 [ERR] OSD full dropping all updates 100% full
146
2017-03-02 16:05:45.129800 osd.5 [ERR] OSD full dropping all updates 100% full
147
2017-03-02 16:05:46.758205 osd.0 [ERR] OSD full dropping all updates 100% full
148
2017-03-02 16:05:40.851872 osd.1 [ERR] OSD full dropping all updates 97% full
149
2017-03-02 16:05:50.965096 mon.0 [INF] pgmap v131: 248 pgs: 248 active+clean; 2168 MB data, 7952 MB us
150
ed, 2738 MB / 10690 MB avail; 11646 kB/s wr, 5 op/s
151
2017-03-02 16:05:50.965347 mon.0 [INF] osdmap e46: 8 osds: 8 up, 8 in
152
2017-03-02 16:05:51.064242 mon.0 [INF] pgmap v132: 248 pgs: 248 active+clean; 2252 MB data, 8746 MB us
153
ed, 1943 MB / 10690 MB avail; 13637 kB/s wr, 6 op/s
154
2017-03-02 16:05:46.285956 osd.6 [WRN] OSD near full (94%)
155
2017-03-02 16:05:52.138187 mon.0 [INF] pgmap v133: 248 pgs: 248 active+clean; 2424 MB data, 9444 MB us
156
ed, 1245 MB / 10690 MB avail; 21955 kB/s wr, 10 op/s
157
2017-03-02 16:05:53.567886 mon.0 [INF] osdmap e47: 8 osds: 8 up, 8 in full
158
2017-03-02 16:05:53.663061 mon.0 [INF] pgmap v134: 248 pgs: 248 active+clean; 2424 MB data, 9444 MB us
159
ed, 1245 MB / 10690 MB avail; 25169 kB/s wr, 12 op/s
160
2017-03-02 16:05:56.310635 mon.0 [INF] pgmap v135: 248 pgs: 248 active+clean; 2424 MB data, 9424 MB us
161
ed, 1265 MB / 10690 MB avail; 25361 kB/s wr, 12 op/s
162
2017-03-02 16:05:57.524510 mon.0 [INF] pgmap v136: 248 pgs: 248 active+clean; 2424 MB data, 9167 MB us
163
ed, 1522 MB / 10690 MB avail; 22673 kB/s wr, 11 op/s
164
2017-03-02 16:05:54.958142 osd.3 [WRN] OSD near full (97%)
165
2017-03-02 16:06:01.309856 mon.0 [INF] pgmap v137: 248 pgs: 248 active+clean; 2424 MB data, 9167 MB us
166
ed, 1523 MB / 10690 MB avail; 18882 kB/s wr, 9 op/s
167
2017-03-02 16:06:02.391238 mon.0 [INF] pgmap v138: 248 pgs: 248 active+clean; 2424 MB data, 9098 MB us
168
ed, 1591 MB / 10690 MB avail; 15223 kB/s wr, 7 op/s
169
2017-03-02 16:06:06.314569 mon.0 [INF] pgmap v139: 248 pgs: 248 active+clean; 2424 MB data, 9098 MB us
170
ed, 1592 MB / 10690 MB avail
171
2017-03-02 16:06:06.646525 mon.0 [INF] HEALTH_ERR; 4 full osd(s); 1 near full osd(s); full flag(s) set
172
2017-03-02 16:06:12.157815 osd.1 [ERR] OSD full dropping all updates 99% full
173
2017-03-02 16:06:16.034446 osd.5 [ERR] OSD full dropping all updates 100% full
174
2017-03-02 16:06:20.663582 osd.0 [ERR] OSD full dropping all updates 98% full
175
2017-03-02 16:06:24.261788 osd.3 [WRN] OSD near full (97%)
176
2017-03-02 16:06:42.363745 osd.1 [ERR] OSD full dropping all updates 99% full
177
2017-03-02 16:06:46.339598 osd.5 [ERR] OSD full dropping all updates 100% full
178
2017-03-02 16:06:53.968147 osd.0 [ERR] OSD full dropping all updates 98% full
179
2017-03-02 16:06:57.469759 osd.3 [WRN] OSD near full (97%)
180
2017-03-02 16:07:13.268817 osd.1 [ERR] OSD full dropping all updates 99% full
181
2017-03-02 16:07:25.978622 osd.0 [ERR] OSD full dropping all updates 98% full
182
2017-03-02 16:07:19.346106 osd.5 [ERR] OSD full dropping all updates 100% full
183
2017-03-02 16:07:31.374800 osd.3 [WRN] OSD near full (97%)
184
2017-03-02 16:07:45.775181 osd.1 [ERR] OSD full dropping all updates 99% full
185
2017-03-02 16:07:55.384920 osd.0 [ERR] OSD full dropping all updates 98% full
186
2017-03-02 16:07:50.951336 osd.5 [ERR] OSD full dropping all updates 100% full
187
2017-03-02 16:08:01.680040 osd.3 [WRN] OSD near full (97%)
188
2017-03-02 16:08:16.581150 osd.1 [ERR] OSD full dropping all updates 99% full
189
2017-03-02 16:08:25.490884 osd.0 [ERR] OSD full dropping all updates 98% full
190
2017-03-02 16:08:21.256363 osd.5 [ERR] OSD full dropping all updates 100% full
191
2017-03-02 16:08:36.085325 osd.3 [WRN] OSD near full (97%)
192
2017-03-02 16:08:46.087673 osd.1 [ERR] OSD full dropping all updates 99% full
193
2017-03-02 16:08:56.599101 osd.0 [ERR] OSD full dropping all updates 98% full
194
2017-03-02 16:08:53.763419 osd.5 [ERR] OSD full dropping all updates 100% full
195
2017-03-02 16:09:18.391436 osd.1 [ERR] OSD full dropping all updates 99% full
196
2017-03-02 16:09:09.791576 osd.3 [WRN] OSD near full (97%)
197
2017-03-02 16:09:23.869551 osd.5 [ERR] OSD full dropping all updates 100% full
198
2017-03-02 16:09:27.305424 osd.0 [ERR] OSD full dropping all updates 98% full
199
2017-03-02 16:09:39.595946 osd.3 [WRN] OSD near full (97%)
200
2017-03-02 16:09:50.496134 osd.1 [ERR] OSD full dropping all updates 99% full
201
2017-03-02 16:09:54.373991 osd.5 [ERR] OSD full dropping all updates 100% full
202
2017-03-02 16:09:58.011731 osd.0 [ERR] OSD full dropping all updates 98% full
203
2017-03-02 16:10:09.201507 osd.3 [WRN] OSD near full (97%)
204
2017-03-02 16:10:22.045770 mon.0 [INF] pgmap v140: 248 pgs: 248 active+clean; 2424 MB data, 8953 MB us
205
ed, 1736 MB / 10690 MB avail
206
2017-03-02 16:10:23.106920 mon.0 [INF] pgmap v141: 248 pgs: 248 active+clean; 2424 MB data, 8779 MB us
207
ed, 1911 MB / 10690 MB avail
208
2017-03-02 16:10:26.881719 osd.5 [ERR] OSD full dropping all updates 100% full
209
2017-03-02 16:10:36.333459 mon.0 [INF] pgmap v142: 248 pgs: 248 active+clean; 2424 MB data, 8459 MB us
210
ed, 2230 MB / 10690 MB avail
211
2017-03-02 16:10:37.451547 mon.0 [INF] pgmap v143: 248 pgs: 248 active+clean; 2424 MB data, 7750 MB us
212
ed, 2939 MB / 10690 MB avail
213
2017-03-02 16:10:38.369779 mon.0 [INF] osdmap e48: 8 osds: 8 up, 8 in nearfull
214
2017-03-02 16:10:38.476568 mon.0 [INF] pgmap v144: 248 pgs: 248 active+clean; 2424 MB data, 7750 MB us
215
ed, 2939 MB / 10690 MB avail
216
2017-03-02 16:10:39.112042 osd.3 [WRN] OSD near full (92%)
217
2017-03-02 16:10:41.412518 osd.3 [ERR] OSD full dropping all updates 100% full
218
2017-03-02 16:10:41.583273 osd.5 [ERR] OSD full dropping all updates 98% full
219
2017-03-02 16:10:43.618776 osd.0 [ERR] OSD full dropping all updates 100% full
220
2017-03-02 16:10:47.433980 mon.0 [INF] pgmap v145: 248 pgs: 248 active+clean; 2456 MB data, 7858 MB us
221
ed, 2831 MB / 10690 MB avail; 119 kB/s wr, 0 op/s
222
2017-03-02 16:10:48.135531 mon.0 [INF] osd.1 172.16.1.24:6800/9261 failed (forced)
223
2017-03-02 16:10:49.504743 mon.0 [INF] pgmap v146: 248 pgs: 248 active+clean; 2700 MB data, 9478 MB us
224
ed, 1211 MB / 10690 MB avail; 10682 kB/s wr, 5 op/s
225
2017-03-02 16:10:49.509016 mon.0 [INF] osdmap e49: 8 osds: 7 up, 8 in nearfull
226
2017-03-02 16:10:49.687370 mon.0 [INF] pgmap v147: 248 pgs: 248 active+clean; 2700 MB data, 9478 MB us
227
ed, 1211 MB / 10690 MB avail; 10802 kB/s wr, 5 op/s
228
2017-03-02 16:10:49.882669 mon.0 [INF] osdmap e50: 8 osds: 7 up, 8 in nearfull
229
2017-03-02 16:10:50.292783 mon.0 [INF] pgmap v148: 248 pgs: 20 stale+active+clean, 228 active+clean; 2
230
700 MB data, 9478 MB used, 1211 MB / 10690 MB avail; 20861 kB/s wr, 10 op/s
231
2017-03-02 16:10:50.942875 mon.0 [INF] osdmap e51: 8 osds: 7 up, 8 in nearfull
232
2017-03-02 16:10:51.129324 mon.0 [INF] pgmap v149: 248 pgs: 20 stale+active+clean, 228 active+clean; 2
233
700 MB data, 9478 MB used, 1211 MB / 10690 MB avail; 20852 kB/s wr, 10 op/s
234
2017-03-02 16:10:45.493219 osd.7 [WRN] OSD near full (91%)
235
2017-03-02 16:10:52.544006 mon.0 [INF] pgmap v150: 248 pgs: 27 active+undersized+degraded, 12 stale+ac
236
tive+clean, 209 active+clean; 2736 MB data, 9535 MB used, 1154 MB / 10690 MB avail; 23165 kB/s wr, 11
237
op/s; 84/2052 objects degraded (4.094%)
238
2017-03-02 16:10:46.238825 osd.6 [WRN] OSD near full (97%)
239
2017-03-02 16:10:48.539306 osd.6 [ERR] OSD full dropping all updates 98% full
240
2017-03-02 16:10:56.631291 mon.0 [INF] pgmap v151: 248 pgs: 87 active+undersized+degraded, 161 active+
241
clean; 2820 MB data, 9710 MB used, 979 MB / 10690 MB avail; 30406 kB/s wr, 14 op/s; 261/2115 objects d
242
egraded (12.340%)
243
2017-03-02 16:10:56.631605 mon.0 [INF] osdmap e52: 8 osds: 7 up, 8 in full
244
2017-03-02 16:10:56.759901 mon.0 [INF] pgmap v152: 248 pgs: 87 active+undersized+degraded, 161 active+
245
clean; 2820 MB data, 9706 MB used, 983 MB / 10690 MB avail; 15048 kB/s wr, 7 op/s; 261/2115 objects de
246
graded (12.340%)
247
2017-03-02 16:10:57.840071 mon.0 [INF] pgmap v153: 248 pgs: 87 active+undersized+degraded, 161 active+
248
clean; 2848 MB data, 9719 MB used, 971 MB / 10690 MB avail; 17720 kB/s wr, 8 op/s; 265/2136 objects de
249
graded (12.406%)
250
2017-03-02 16:11:01.376789 mon.0 [INF] pgmap v154: 248 pgs: 87 active+undersized+degraded, 161 active+
251
clean; 2848 MB data, 9718 MB used, 971 MB / 10690 MB avail; 13223 kB/s wr, 6 op/s; 265/2136 objects de
252
graded (12.406%)
253
2017-03-02 16:10:56.740054 osd.6 [WRN] OSD near full (92%)
254
2017-03-02 16:11:02.545336 mon.0 [INF] pgmap v155: 248 pgs: 87 active+undersized+degraded, 161 active+
255
clean; 2848 MB data, 9468 MB used, 1221 MB / 10690 MB avail; 13183 kB/s wr, 6 op/s; 265/2136 objects d
256
egraded (12.406%)
257
2017-03-02 16:11:06.647816 mon.0 [INF] HEALTH_ERR; 87 pgs degraded; 34 pgs stuck unclean; 87 pgs under
258
sized; recovery 265/2136 objects degraded (12.406%); 3 full osd(s); 3 near full osd(s); 1/8 in osds ar
259
e down; full flag(s) set
260
2017-03-02 16:11:07.865842 mon.0 [INF] pgmap v156: 248 pgs: 87 active+undersized+degraded, 161 active+
261
clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 7327 kB/s wr, 3 op/s; 265/2136 objects de
262
graded (12.406%)
263
2017-03-02 16:11:09.780896 osd.1 [ERR] OSD full dropping all updates 100% full
264
2017-03-02 16:11:09.786239 mon.0 [INF] from='osd.1 172.16.1.24:6800/10572' entity='osd.1' cmd=[{"prefi
265
x": "osd crush create-or-move", "id": 1, "weight":0.0010, "args": ["host=data4", "root=default"]}]: di
266
spatch
267
2017-03-02 16:11:10.120022 mon.0 [INF] osd.1 172.16.1.24:6800/10572 boot
268
2017-03-02 16:11:10.192750 mon.0 [INF] osdmap e53: 8 osds: 8 up, 8 in full
269
2017-03-02 16:11:10.305990 mon.0 [INF] pgmap v157: 248 pgs: 87 active+undersized+degraded, 161 active+
270
clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 1729 kB/s wr, 0 op/s; 265/2136 objects de
271
graded (12.406%)
272
2017-03-02 16:11:10.489964 mon.0 [INF] osdmap e54: 8 osds: 8 up, 8 in full
273
2017-03-02 16:11:10.547510 mon.0 [INF] pgmap v158: 248 pgs: 87 active+undersized+degraded, 161 active+
274
clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 2071 kB/s wr, 1 op/s; 265/2136 objects de
275
graded (12.406%)
276
2017-03-02 16:11:11.604070 mon.0 [INF] osdmap e55: 8 osds: 8 up, 8 in full
277
2017-03-02 16:11:11.743501 mon.0 [INF] pgmap v159: 248 pgs: 5 peering, 1 remapped, 81 active+undersize
278
d+degraded, 161 active+clean; 2848 MB data, 9468 MB used, 1222 MB / 10690 MB avail; 241/2136 objects d
279
egraded (11.283%)
280
2017-03-02 16:11:11.947407 mon.0 [INF] osdmap e56: 8 osds: 8 up, 8 in full
281
2017-03-02 16:11:12.277552 mon.0 [INF] pgmap v160: 248 pgs: 8 activating, 24 peering, 1 remapped, 53 a
282
ctive+undersized+degraded, 1 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229 M
283
B / 10690 MB avail; 162/2136 objects degraded (7.584%)
284
2017-03-02 16:11:13.062951 mon.0 [INF] osdmap e57: 8 osds: 8 up, 8 in full
285
2017-03-02 16:11:13.111860 mon.0 [INF] osd.1 172.16.1.24:6800/10572 failed (forced)
286
2017-03-02 16:11:13.148836 mon.0 [INF] pgmap v161: 248 pgs: 20 activating, 40 peering, 1 remapped, 20
287
active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229
288
MB / 10690 MB avail; 63/2136 objects degraded (2.949%)
289
2017-03-02 16:11:14.224893 mon.0 [INF] osdmap e58: 8 osds: 7 up, 8 in full
290
2017-03-02 16:11:14.310806 mon.0 [INF] pgmap v162: 248 pgs: 20 activating, 40 peering, 1 remapped, 20
291
active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229
292
MB / 10690 MB avail; 63/2136 objects degraded (2.949%)
293
2017-03-02 16:11:14.518233 mon.0 [INF] osdmap e59: 8 osds: 7 up, 8 in full
294
2017-03-02 16:11:14.606351 mon.0 [INF] pgmap v163: 248 pgs: 20 activating, 40 peering, 1 remapped, 20
295
active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229
296
MB / 10690 MB avail; 63/2136 objects degraded (2.949%)
297
2017-03-02 16:11:15.569987 mon.0 [INF] osdmap e60: 8 osds: 7 up, 8 in full
298
2017-03-02 16:11:15.702331 mon.0 [INF] pgmap v164: 248 pgs: 20 activating, 40 peering, 1 remapped, 20
299
active+undersized+degraded, 6 activating+degraded, 161 active+clean; 2848 MB data, 9460 MB used, 1229
300
MB / 10690 MB avail; 63/2136 objects degraded (2.949%)
301
2017-03-02 16:11:12.921904 osd.0 [WRN] OSD near full (97%)
302
2017-03-02 16:11:16.790273 mon.0 [INF] pgmap v165: 248 pgs: 12 activating, 16 peering, 54 active+under
303
sized+degraded, 5 activating+degraded, 161 active+clean; 2848 MB data, 9416 MB used, 1273 MB / 10690 M
304
B avail; 162/2136 objects degraded (7.584%); 795 kB/s, 0 objects/s recovering
305
2017-03-02 16:11:17.871439 mon.0 [INF] pgmap v166: 248 pgs: 87 active+undersized+degraded, 161 active+
306
clean; 2848 MB data, 9383 MB used, 1307 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 695
307
kB/s, 0 objects/s recovering
308
2017-03-02 16:11:12.618251 osd.3 [ERR] OSD full dropping all updates 100% full
309
2017-03-02 16:11:14.786176 osd.5 [ERR] OSD full dropping all updates 99% full
310
2017-03-02 16:11:21.374454 mon.0 [INF] pgmap v167: 248 pgs: 87 active+undersized+degraded, 161 active+
311
clean; 2848 MB data, 9382 MB used, 1307 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 493
312
kB/s, 0 objects/s recovering
313
2017-03-02 16:11:22.537417 mon.0 [INF] pgmap v168: 248 pgs: 87 active+undersized+degraded, 161 active+
314
clean; 2848 MB data, 9380 MB used, 1309 MB / 10690 MB avail; 255/2136 objects degraded (11.938%); 497
315
kB/s, 0 objects/s recovering
316
2017-03-02 16:11:26.845284 osd.6 [WRN] OSD near full (90%)
317
2017-03-02 16:11:44.727636 osd.0 [WRN] OSD near full (97%)
318
2017-03-02 16:11:44.688298 osd.5 [ERR] OSD full dropping all updates 99% full
319
2017-03-02 16:11:47.722370 osd.3 [ERR] OSD full dropping all updates 100% full
320
2017-03-02 16:11:57.750631 osd.6 [WRN] OSD near full (90%)
321
2017-03-02 16:12:06.648141 mon.0 [INF] HEALTH_ERR; 87 pgs degraded; 21 pgs stuck unclean; 87 pgs under
322
sized; recovery 255/2136 objects degraded (11.938%); 3 full osd(s); 3 near full osd(s); 1/8 in osds ar
323
e down; full flag(s) set
324
2017-03-02 16:12:14.531442 osd.0 [WRN] OSD near full (97%)
325
2017-03-02 16:12:17.526405 osd.3 [ERR] OSD full dropping all updates 100% full
326
2017-03-02 16:12:18.091959 osd.5 [ERR] OSD full dropping all updates 99% full
327
2017-03-02 16:12:28.754698 osd.6 [WRN] OSD near full (90%)
328
2017-03-02 16:12:44.935946 osd.0 [WRN] OSD near full (97%)
329
2017-03-02 16:12:47.231282 osd.3 [ERR] OSD full dropping all updates 100% full
330
2017-03-02 16:12:50.597548 osd.5 [ERR] OSD full dropping all updates 99% full
331
2017-03-02 16:13:02.960850 osd.6 [WRN] OSD near full (90%)
332
2017-03-02 16:13:15.241017 osd.0 [WRN] OSD near full (97%)
333
2017-03-02 16:13:18.235574 osd.3 [ERR] OSD full dropping all updates 100% full
334
2017-03-02 16:13:21.101305 osd.5 [ERR] OSD full dropping all updates 99% full
335
2017-03-02 16:13:34.766836 osd.6 [WRN] OSD near full (90%)
336
2017-03-02 16:13:47.246341 osd.0 [WRN] OSD near full (97%)
337
2017-03-02 16:13:50.041517 osd.3 [ERR] OSD full dropping all updates 100% full
338
2017-03-02 16:13:53.105995 osd.5 [ERR] OSD full dropping all updates 99% full
339
2017-03-02 16:14:05.472628 osd.6 [WRN] OSD near full (90%)
340
2017-03-02 16:14:17.451810 osd.0 [WRN] OSD near full (97%)
341
2017-03-02 16:14:22.646883 osd.3 [ERR] OSD full dropping all updates 100% full
342
2017-03-02 16:14:24.710631 osd.5 [ERR] OSD full dropping all updates 99% full
343
2017-03-02 16:14:35.777537 osd.6 [WRN] OSD near full (90%)
344
2017-03-02 16:14:50.856136 osd.0 [WRN] OSD near full (97%)
345
2017-03-02 16:14:56.853288 osd.3 [ERR] OSD full dropping all updates 100% full
346
2017-03-02 16:14:57.017252 osd.5 [ERR] OSD full dropping all updates 99% full
347
2017-03-02 16:15:06.236347 7fd792ffd700  0 -- 172.16.1.254:0/1637311347 >> 172.16.1.11:6800/6899 conn(
348
0x7fd780009fb0 :-1 s=STATE_OPEN pgs=837 cs=1 l=0).fault initiating reconnect
349
^Cadmin:~ # ceph -s
350
    cluster 557c4525-cad3-371a-8e05-4b95f6accdf5
351
     health HEALTH_ERR
352
            87 pgs degraded
353
            21 pgs stuck unclean
354
            87 pgs undersized
355
            recovery 255/2136 objects degraded (11.938%)
356
            3 full osd(s)
357
            3 near full osd(s)
358
            1/8 in osds are down
359
            full flag(s) set
360
     monmap e2: 3 mons at {data1=172.16.1.21:6789/0,data2=172.16.1.22:6789/0,mon1=172.16.1.11:6789/0}
361
            election epoch 6, quorum 0,1,2 mon1,data1,data2
362
        mgr active: mon1 standbys: data1, data2
363
     osdmap e60: 8 osds: 7 up, 8 in; 66 remapped pgs
364
            flags nearfull,full,sortbitwise,require_jewel_osds,require_kraken_osds,require_luminous_os
365
ds
366
      pgmap v168: 248 pgs, 1 pools, 2848 MB data, 712 objects
367
            9380 MB used, 1309 MB / 10690 MB avail
368
            255/2136 objects degraded (11.938%)
369
                 161 active+clean
370
                  87 active+undersized+degraded
371
admin:~ # ceph osd tree
372
ID WEIGHT  TYPE NAME      UP/DOWN REWEIGHT PRIMARY-AFFINITY
373
-1 0.01035 root default
374
-2 0.00198     host data3
375
 0 0.00099         osd.0       up  1.00000          1.00000
376
 3 0.00099         osd.3       up  1.00000          1.00000
377
-3 0.00339     host data2
378
 2 0.00169         osd.2       up  1.00000          1.00000
379
 4 0.00169         osd.4       up  1.00000          1.00000
380
-4 0.00198     host data4
381
 1 0.00099         osd.1     down  1.00000          1.00000
382
 5 0.00099         osd.5       up  1.00000          1.00000
383
-5 0.00299     host data1
384
 6 0.00150         osd.6       up  1.00000          1.00000
385
 7 0.00150         osd.7       up  1.00000          1.00000
386
admin:~ #