https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2019-02-06T15:32:42ZCeph CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1288792019-02-06T15:32:42ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Patrick Donnelly</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li><li><strong>Target version</strong> set to <i>v14.0.0</i></li><li><strong>Start date</strong> deleted (<del><i>02/06/2019</i></del>)</li><li><strong>Backport</strong> set to <i>mimic,luminous</i></li><li><strong>Affected Versions</strong> <i>v12.2.10</i> added</li></ul> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1291352019-02-12T03:41:14ZZheng Yanukernel@gmail.com
<ul></ul><p>please gdb the coredump, check where is MDRequestImpl::~MDRequestImpl()+0x224</p> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1291532019-02-12T09:21:45ZDan van der Ster
<ul></ul><pre>
(gdb) up
#12 0x0000561d9f1b5b64 in MDRequestImpl::~MDRequestImpl (this=0x561fdc42b700, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.10/src/mds/Mutation.cc:189
189 delete _more;
(gdb) list
184 {
185 if (client_request)
186 client_request->put();
187 if (slave_request)
188 slave_request->put();
189 delete _more;
190 }
191
192 MDRequestImpl::More* MDRequestImpl::more()
193 {
(gdb) p _more
$1 = (MDRequestImpl::More *) 0x561df4e52000
(gdb)
</pre> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1291782019-02-12T14:26:16ZDan van der Ster
<ul></ul><p>up a bit higher:<br /><pre>
#14 0x0000561d9f3a88e2 in put (this=<optimized out>) at /usr/src/debug/ceph-12.2.10/src/common/TrackedOp.h:236
236 delete this;
(gdb) list
231 void put() {
232 if (--nref == 0) {
233 switch (state.load()) {
234 case STATE_UNTRACKED:
235 _unregistered();
236 delete this;
237 break;
238
239 case STATE_LIVE:
240 mark_event("done");
(gdb)
(gdb) up
#15 intrusive_ptr_release (o=<optimized out>) at /usr/src/debug/ceph-12.2.10/src/common/TrackedOp.h:315
315 o->put();
(gdb) list
310 // put for historical op tracking
311 friend void intrusive_ptr_add_ref(TrackedOp *o) {
312 o->get();
313 }
314 friend void intrusive_ptr_release(TrackedOp *o) {
315 o->put();
316 }
317 };
318
319
(gdb) up
#16 ~intrusive_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.10/build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:98
98 if( px != 0 ) intrusive_ptr_release( px );
(gdb) up
#17 OpTracker::unregister_inflight_op (this=0x561da9e172e8, i=<optimized out>) at /usr/src/debug/ceph-12.2.10/src/common/TrackedOp.cc:283
283 history.insert(now, TrackedOpRef(i));
(gdb) list
278 if (!tracking_enabled)
279 delete i;
280 else {
281 i->state = TrackedOp::STATE_HISTORY;
282 utime_t now = ceph_clock_now();
283 history.insert(now, TrackedOpRef(i));
284 }
285 }
286
287 bool OpTracker::check_ops_in_flight(std::vector<string> &warning_vector, int *slow)
(gdb)
(gdb) up
#18 0x0000561d9f2c9196 in intrusive_ptr_release (o=<optimized out>) at /usr/src/debug/ceph-12.2.10/src/common/TrackedOp.h:315
315 o->put();
(gdb) up
#19 ~intrusive_ptr (this=0x7f7be8bd2f10, __in_chrg=<optimized out>) at /usr/src/debug/ceph-12.2.10/build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:98
98 if( px != 0 ) intrusive_ptr_release( px );
(gdb) up
#20 Migrator::handle_export_discover_ack (this=this@entry=0x561da9e36630, m=m@entry=0x5620ade63e00) at /usr/src/debug/ceph-12.2.10/src/mds/Migrator.cc:1320
1320 assert(g_conf->mds_kill_export_at != 3);
(gdb) up
#21 0x0000561d9f2d370b in Migrator::dispatch (this=0x561da9e36630, m=m@entry=0x5620ade63e00) at /usr/src/debug/ceph-12.2.10/src/mds/Migrator.cc:146
146 handle_export_discover_ack(static_cast<MExportDirDiscoverAck*>(m));
</pre></p> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1292652019-02-13T08:54:53ZZheng Yanukernel@gmail.com
<ul></ul><p>In MDRequestImpl::~MDRequestImpl, please check if 'state' is STATE_UNTRACKED.</p>
<p>In Migrator::handle_export_discover_ack, please check if 'mdr.px' is 0x561fdc42b700. (if mdr is optimized out, go down a level to ~intrusive_ptr, check px)</p> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1292692019-02-13T09:43:06ZDan van der Ster
<ul></ul><pre>
(gdb) f 12
#12 0x0000561d9f1b5b64 in MDRequestImpl::~MDRequestImpl (this=0x561fdc42b700, __in_chrg=<optimized out>)
at /usr/src/debug/ceph-12.2.10/src/mds/Mutation.cc:189
189 delete _more;
(gdb) p state
$1 = {<std::__atomic_base<int>> = {_M_i = 2}, <No data fields>}
(gdb) f 20
#20 Migrator::handle_export_discover_ack (this=this@entry=0x561da9e36630, m=m@entry=0x5620ade63e00)
at /usr/src/debug/ceph-12.2.10/src/mds/Migrator.cc:1320
1320 assert(g_conf->mds_kill_export_at != 3);
(gdb) p mdr.px
$2 = (MDRequestImpl *) 0x561fdc42b700
(gdb)
</pre> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1292962019-02-13T13:09:33ZZheng Yanukernel@gmail.com
<ul></ul><p>Thanks. But everything looks fine. I have no idea what happened</p> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1292972019-02-13T13:13:16ZDan van der Ster
<ul></ul><p>OK, we only had this type of crash once or twice, so feel free to drop the priority.<br />We'll let you know if it reoccurs.</p> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1310382019-03-07T23:22:22ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> changed from <i>v14.0.0</i> to <i>v15.0.0</i></li></ul> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1314172019-03-09T00:32:21ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Category</strong> deleted (<del><i>90</i></del>)</li></ul> CephFS - Bug #38203: ceph-mds segfault during migrator nicely exportinghttps://tracker.ceph.com/issues/38203?journal_id=1566192020-01-23T01:15:05ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Can't reproduce</i></li><li><strong>Assignee</strong> deleted (<del><i>Patrick Donnelly</i></del>)</li><li><strong>Target version</strong> deleted (<del><i>v15.0.0</i></del>)</li><li><strong>Backport</strong> deleted (<del><i>mimic,luminous</i></del>)</li></ul>