https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-02-25T17:00:33ZCeph bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1079982018-02-25T17:00:33ZOliver Freyermuth
<ul></ul><p>It might be that this OSD was subject to OOM at some point in the last 24 hours. <br />It seems OSDs are using 2-3 times as much memory as configured via bluestore_cache_size_hdd when doing recovery and accepting small objects,<br />which exceeded our RAM + swap on some machines.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1079992018-02-25T17:09:09ZOliver Freyermuth
<ul></ul><p>Here's another log of another OSD:<br />7de1dddf-27d4-4b6b-9128-0138bfaf85cf<br />backtrace looks similar.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080002018-02-25T17:26:12ZOliver Freyermuth
<ul></ul><p>After many restarts of all OSDs, and temporarily lowering min_size, they now stay up. I'll watch and see if the cluster recovers.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080022018-02-25T19:27:13ZOliver Freyermuth
<ul></ul><p>Cluster has mostly recovered, looks good. <br />Still, hopefully the stacktrace and logs can help to track down the underlying issue that caused the crashes.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080032018-02-25T23:51:04ZOliver Freyermuth
<ul></ul><p>Here's a ceph osd tree due to popular request:<br /><pre>
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 700.74890 root default
-3 0.43658 host mon001
0 ssd 0.21829 osd.0 up 1.00000 1.00000
1 ssd 0.21829 osd.1 up 1.00000 1.00000
-5 0.43637 host mon002
2 ssd 0.21819 osd.2 up 1.00000 1.00000
3 ssd 0.21819 osd.3 up 1.00000 1.00000
-10 116.64600 host osd001
4 hdd 3.64519 osd.4 up 1.00000 1.00000
5 hdd 3.64519 osd.5 up 1.00000 1.00000
6 hdd 3.64519 osd.6 up 1.00000 1.00000
7 hdd 3.64519 osd.7 up 1.00000 1.00000
8 hdd 3.64519 osd.8 up 1.00000 1.00000
9 hdd 3.64519 osd.9 up 1.00000 1.00000
10 hdd 3.64519 osd.10 up 1.00000 1.00000
11 hdd 3.64519 osd.11 up 1.00000 1.00000
12 hdd 3.64519 osd.12 up 1.00000 1.00000
13 hdd 3.64519 osd.13 up 1.00000 1.00000
14 hdd 3.64519 osd.14 up 1.00000 1.00000
15 hdd 3.64519 osd.15 up 1.00000 1.00000
16 hdd 3.64519 osd.16 up 1.00000 1.00000
17 hdd 3.64519 osd.17 up 1.00000 1.00000
18 hdd 3.64519 osd.18 up 1.00000 1.00000
19 hdd 3.64519 osd.19 up 1.00000 1.00000
20 hdd 3.64519 osd.20 up 1.00000 1.00000
21 hdd 3.64519 osd.21 up 1.00000 1.00000
22 hdd 3.64519 osd.22 up 1.00000 1.00000
23 hdd 3.64519 osd.23 up 1.00000 1.00000
24 hdd 3.64519 osd.24 up 1.00000 1.00000
25 hdd 3.64519 osd.25 up 1.00000 1.00000
26 hdd 3.64519 osd.26 up 1.00000 1.00000
27 hdd 3.64519 osd.27 up 1.00000 1.00000
28 hdd 3.64519 osd.28 up 1.00000 1.00000
29 hdd 3.64519 osd.29 up 1.00000 1.00000
30 hdd 3.64519 osd.30 up 1.00000 1.00000
31 hdd 3.64519 osd.31 up 1.00000 1.00000
32 hdd 3.64519 osd.32 up 1.00000 1.00000
33 hdd 3.64519 osd.33 up 1.00000 1.00000
34 hdd 3.64519 osd.34 up 1.00000 1.00000
35 hdd 3.64519 osd.35 up 1.00000 1.00000
-13 116.64600 host osd002
36 hdd 3.64519 osd.36 up 1.00000 1.00000
37 hdd 3.64519 osd.37 up 1.00000 1.00000
38 hdd 3.64519 osd.38 up 1.00000 1.00000
39 hdd 3.64519 osd.39 up 1.00000 1.00000
40 hdd 3.64519 osd.40 up 1.00000 1.00000
41 hdd 3.64519 osd.41 up 1.00000 1.00000
42 hdd 3.64519 osd.42 up 1.00000 1.00000
43 hdd 3.64519 osd.43 up 1.00000 1.00000
44 hdd 3.64519 osd.44 up 1.00000 1.00000
45 hdd 3.64519 osd.45 up 1.00000 1.00000
46 hdd 3.64519 osd.46 up 1.00000 1.00000
47 hdd 3.64519 osd.47 up 1.00000 1.00000
48 hdd 3.64519 osd.48 up 1.00000 1.00000
49 hdd 3.64519 osd.49 up 1.00000 1.00000
50 hdd 3.64519 osd.50 up 1.00000 1.00000
51 hdd 3.64519 osd.51 up 1.00000 1.00000
52 hdd 3.64519 osd.52 up 1.00000 1.00000
53 hdd 3.64519 osd.53 up 1.00000 1.00000
54 hdd 3.64519 osd.54 up 1.00000 1.00000
55 hdd 3.64519 osd.55 up 1.00000 1.00000
56 hdd 3.64519 osd.56 up 1.00000 1.00000
57 hdd 3.64519 osd.57 up 1.00000 1.00000
58 hdd 3.64519 osd.58 up 1.00000 1.00000
59 hdd 3.64519 osd.59 up 1.00000 1.00000
60 hdd 3.64519 osd.60 up 1.00000 1.00000
61 hdd 3.64519 osd.61 up 1.00000 1.00000
62 hdd 3.64519 osd.62 up 1.00000 1.00000
63 hdd 3.64519 osd.63 up 1.00000 1.00000
64 hdd 3.64519 osd.64 up 1.00000 1.00000
65 hdd 3.64519 osd.65 up 1.00000 1.00000
66 hdd 3.64519 osd.66 up 1.00000 1.00000
67 hdd 3.64519 osd.67 up 1.00000 1.00000
-16 116.64600 host osd003
68 hdd 3.64519 osd.68 up 1.00000 1.00000
69 hdd 3.64519 osd.69 up 1.00000 1.00000
70 hdd 3.64519 osd.70 up 1.00000 1.00000
71 hdd 3.64519 osd.71 up 1.00000 1.00000
72 hdd 3.64519 osd.72 up 1.00000 1.00000
73 hdd 3.64519 osd.73 up 1.00000 1.00000
74 hdd 3.64519 osd.74 up 1.00000 1.00000
75 hdd 3.64519 osd.75 up 1.00000 1.00000
76 hdd 3.64519 osd.76 up 1.00000 1.00000
77 hdd 3.64519 osd.77 up 1.00000 1.00000
78 hdd 3.64519 osd.78 up 1.00000 1.00000
79 hdd 3.64519 osd.79 up 1.00000 1.00000
80 hdd 3.64519 osd.80 up 1.00000 1.00000
81 hdd 3.64519 osd.81 up 1.00000 1.00000
82 hdd 3.64519 osd.82 up 1.00000 1.00000
83 hdd 3.64519 osd.83 up 1.00000 1.00000
84 hdd 3.64519 osd.84 up 1.00000 1.00000
85 hdd 3.64519 osd.85 up 1.00000 1.00000
86 hdd 3.64519 osd.86 up 1.00000 1.00000
87 hdd 3.64519 osd.87 up 1.00000 1.00000
88 hdd 3.64519 osd.88 up 1.00000 1.00000
89 hdd 3.64519 osd.89 up 1.00000 1.00000
90 hdd 3.64519 osd.90 up 1.00000 1.00000
91 hdd 3.64519 osd.91 up 1.00000 1.00000
92 hdd 3.64519 osd.92 up 1.00000 1.00000
93 hdd 3.64519 osd.93 up 1.00000 1.00000
94 hdd 3.64519 osd.94 up 1.00000 1.00000
95 hdd 3.64519 osd.95 up 1.00000 1.00000
96 hdd 3.64519 osd.96 up 1.00000 1.00000
97 hdd 3.64519 osd.97 up 1.00000 1.00000
98 hdd 3.64519 osd.98 up 1.00000 1.00000
99 hdd 3.64519 osd.99 up 1.00000 1.00000
-19 116.64600 host osd004
100 hdd 3.64519 osd.100 up 1.00000 1.00000
101 hdd 3.64519 osd.101 up 1.00000 1.00000
102 hdd 3.64519 osd.102 up 1.00000 1.00000
103 hdd 3.64519 osd.103 up 1.00000 1.00000
104 hdd 3.64519 osd.104 up 1.00000 1.00000
105 hdd 3.64519 osd.105 up 1.00000 1.00000
106 hdd 3.64519 osd.106 up 1.00000 1.00000
107 hdd 3.64519 osd.107 up 1.00000 1.00000
108 hdd 3.64519 osd.108 up 1.00000 1.00000
109 hdd 3.64519 osd.109 up 1.00000 1.00000
110 hdd 3.64519 osd.110 up 1.00000 1.00000
111 hdd 3.64519 osd.111 up 1.00000 1.00000
112 hdd 3.64519 osd.112 up 1.00000 1.00000
113 hdd 3.64519 osd.113 up 1.00000 1.00000
114 hdd 3.64519 osd.114 up 1.00000 1.00000
115 hdd 3.64519 osd.115 up 1.00000 1.00000
116 hdd 3.64519 osd.116 up 1.00000 1.00000
117 hdd 3.64519 osd.117 up 1.00000 1.00000
118 hdd 3.64519 osd.118 up 1.00000 1.00000
119 hdd 3.64519 osd.119 up 1.00000 1.00000
120 hdd 3.64519 osd.120 up 1.00000 1.00000
121 hdd 3.64519 osd.121 up 1.00000 1.00000
122 hdd 3.64519 osd.122 up 1.00000 1.00000
123 hdd 3.64519 osd.123 up 1.00000 1.00000
124 hdd 3.64519 osd.124 up 1.00000 1.00000
125 hdd 3.64519 osd.125 up 1.00000 1.00000
126 hdd 3.64519 osd.126 up 1.00000 1.00000
127 hdd 3.64519 osd.127 up 1.00000 1.00000
128 hdd 3.64519 osd.128 up 1.00000 1.00000
129 hdd 3.64519 osd.129 up 1.00000 1.00000
130 hdd 3.64519 osd.130 up 1.00000 1.00000
131 hdd 3.64519 osd.131 up 1.00000 1.00000
-22 116.64600 host osd005
132 hdd 3.64519 osd.132 up 1.00000 1.00000
133 hdd 3.64519 osd.133 up 1.00000 1.00000
134 hdd 3.64519 osd.134 up 1.00000 1.00000
135 hdd 3.64519 osd.135 up 1.00000 1.00000
136 hdd 3.64519 osd.136 up 1.00000 1.00000
137 hdd 3.64519 osd.137 up 1.00000 1.00000
138 hdd 3.64519 osd.138 up 1.00000 1.00000
139 hdd 3.64519 osd.139 up 1.00000 1.00000
140 hdd 3.64519 osd.140 up 1.00000 1.00000
141 hdd 3.64519 osd.141 up 1.00000 1.00000
142 hdd 3.64519 osd.142 up 1.00000 1.00000
143 hdd 3.64519 osd.143 up 1.00000 1.00000
144 hdd 3.64519 osd.144 up 1.00000 1.00000
145 hdd 3.64519 osd.145 up 1.00000 1.00000
146 hdd 3.64519 osd.146 up 1.00000 1.00000
147 hdd 3.64519 osd.147 up 1.00000 1.00000
148 hdd 3.64519 osd.148 up 1.00000 1.00000
149 hdd 3.64519 osd.149 up 1.00000 1.00000
150 hdd 3.64519 osd.150 up 1.00000 1.00000
151 hdd 3.64519 osd.151 up 1.00000 1.00000
152 hdd 3.64519 osd.152 up 1.00000 1.00000
153 hdd 3.64519 osd.153 up 1.00000 1.00000
154 hdd 3.64519 osd.154 up 1.00000 1.00000
155 hdd 3.64519 osd.155 up 1.00000 1.00000
156 hdd 3.64519 osd.156 up 1.00000 1.00000
157 hdd 3.64519 osd.157 up 1.00000 1.00000
158 hdd 3.64519 osd.158 up 1.00000 1.00000
159 hdd 3.64519 osd.159 up 1.00000 1.00000
160 hdd 3.64519 osd.160 up 1.00000 1.00000
161 hdd 3.64519 osd.161 up 1.00000 1.00000
162 hdd 3.64519 osd.162 up 1.00000 1.00000
163 hdd 3.64519 osd.163 up 1.00000 1.00000
-25 116.64600 host osd006
164 hdd 3.64519 osd.164 up 1.00000 1.00000
165 hdd 3.64519 osd.165 up 1.00000 1.00000
166 hdd 3.64519 osd.166 up 1.00000 1.00000
167 hdd 3.64519 osd.167 up 1.00000 1.00000
168 hdd 3.64519 osd.168 up 1.00000 1.00000
169 hdd 3.64519 osd.169 up 1.00000 1.00000
170 hdd 3.64519 osd.170 up 1.00000 1.00000
171 hdd 3.64519 osd.171 up 1.00000 1.00000
172 hdd 3.64519 osd.172 up 1.00000 1.00000
173 hdd 3.64519 osd.173 up 1.00000 1.00000
174 hdd 3.64519 osd.174 up 1.00000 1.00000
175 hdd 3.64519 osd.175 up 1.00000 1.00000
176 hdd 3.64519 osd.176 up 1.00000 1.00000
177 hdd 3.64519 osd.177 up 1.00000 1.00000
178 hdd 3.64519 osd.178 up 1.00000 1.00000
179 hdd 3.64519 osd.179 up 1.00000 1.00000
180 hdd 3.64519 osd.180 up 1.00000 1.00000
181 hdd 3.64519 osd.181 up 1.00000 1.00000
182 hdd 3.64519 osd.182 up 1.00000 1.00000
183 hdd 3.64519 osd.183 up 1.00000 1.00000
184 hdd 3.64519 osd.184 up 1.00000 1.00000
185 hdd 3.64519 osd.185 up 1.00000 1.00000
186 hdd 3.64519 osd.186 up 1.00000 1.00000
187 hdd 3.64519 osd.187 up 1.00000 1.00000
188 hdd 3.64519 osd.188 up 1.00000 1.00000
189 hdd 3.64519 osd.189 up 1.00000 1.00000
190 hdd 3.64519 osd.190 up 1.00000 1.00000
191 hdd 3.64519 osd.191 up 1.00000 1.00000
192 hdd 3.64519 osd.192 up 1.00000 1.00000
193 hdd 3.64519 osd.193 up 1.00000 1.00000
194 hdd 3.64519 osd.194 up 1.00000 1.00000
195 hdd 3.64519 osd.195 up 1.00000 1.00000
</pre><br />The metadata-pool lives on the SSDs, the data pool on the HDDs (via the device classes).</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080042018-02-25T23:55:38ZOliver Freyermuth
<ul></ul><p>All HDD-OSDs have 4 TB, while the SSDs used for the metadata pool have 240 GB.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080072018-02-26T02:47:04ZPeter Woodmanpeter@shortbus.org
<ul></ul><p>Hey, I might be seeing the same bug. Can you paste in the operation dump that shows up right before that crash, and maybe like ~10 lines of previous context?</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080142018-02-26T08:56:01ZOliver Freyermuth
<ul></ul><p>@Peter Woodman: Since the system recovered after many OSD restarts (see my previous comment) and I did not think about taking an ops dump, I can sadly not reproduce that now :-(. I'll keep it in mind as soon as the issue reappears, right now I can sadly only share the logfiles (which I uploaded in full via ceph-post-file, but I could also share parts of the publicly if it helps).</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080492018-02-26T19:05:48ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>bluestore</i></li></ul> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080812018-02-26T21:03:32ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul><p>Can you reproduce the crash on one or more OSDs with 'debug osd = 20' and 'debug bluestore = 20'?</p>
<p>Also, can you check what the problematic PG is on the other crashing OSDs? For the attached log (osd.191) it is<br /><pre>
-202> 2018-02-25 17:21:58.829663 7f503e854700 0 bluestore(/var/lib/ceph/osd/ceph-191) transaction dump:
{
"ops": [
{
"op_num": 0,
"op_name": "setattrs",
"collection": "2.21es1_head",
"oid": "1#2:785f4b65:::1000645413e.00000000:head#",
"attr_lens": {
"_": 275,
"_layout": 30,
"_parent": 346,
"snapset": 35
}
},
{
"op_num": 1,
"op_name": "setattr",
"collection": "2.21es1_head",
"oid": "1#2:785f4b65:::1000645413e.00000000:head#",
"name": "hinfo_key",
"length": 42
}
]
}
</pre><br />so 2.21es1</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1080842018-02-26T21:33:43ZOliver Freyermuth
<ul></ul><p>The bad news (for the ticket) is that the problem vanished after restarting all crashing OSDs often enough,<br />and temporarily reducing min_size of the cluster (k=4 m=2, min_size is usually 5, I temporarily put it to 4). <br />So currently, I can't reproduce :-(. <br />Maybe Peter Woodman can add more info if his bug is really the same - <br />Or maybe 7de1dddf-27d4-4b6b-9128-0138bfaf85cf helps (from my comment <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:2178 (Closed)" href="https://tracker.ceph.com/issues/2">#2</a>), which was from another crashing OSD?</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1081052018-02-26T23:21:34ZPeter Woodmanpeter@shortbus.org
<ul></ul><p>Yeah, I've got some of that. Problem is, I'm not seeing debug log messages that should be there based on the failure, if I'm reading the code correctly. Will post when i get home.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1081062018-02-26T23:22:50ZPeter Woodmanpeter@shortbus.org
<ul></ul><p>Actually, looks like your crashing ops are different from mine. I'll just open a new bug.</p> bluestore - Bug #23120: OSDs continously crash during recoveryhttps://tracker.ceph.com/issues/23120?journal_id=1254502018-11-29T15:45:35ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Can't reproduce</i></li></ul>