https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2016-07-22T13:11:39Z
Ceph
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75192
2016-07-22T13:11:39Z
John Spray
jcspray@gmail.com
<ul><li><strong>Component(FS)</strong> <i>MDS</i> added</li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75268
2016-07-24T21:03:46Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Another of the same:</p>
<pre>
Dead: 2016-07-24T02:04:40.579 INFO:tasks.workunit.client.0.mira082.stderr:/home/ubuntu/cephtest/workunit.client.0/suites/fsstress.sh: line 1: 14917 Terminated $command
ceph version v11.0.0-820-ga0294e6 (a0294e64507a7916fdd9707ae22ba40b0d7b65d1)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x91be5b]
2: (CDir::check_rstats(bool)+0x14a6) [0x7a8176]
3: (MDCache::predirty_journal_parents(std::shared_ptr<MutationImpl>, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xf73) [0x6fe913]
4: (Server::_rename_prepare(std::shared_ptr<MDRequestImpl>&, EMetaBlob*, ceph::buffer::list*, CDentry*, CDentry*, CDentry*)+0x6b9) [0x659579]
5: (Server::handle_slave_rename_prep(std::shared_ptr<MDRequestImpl>&)+0xffb) [0x6653fb]
6: (Server::dispatch_slave_request(std::shared_ptr<MDRequestImpl>&)+0x70b) [0x666dab]
7: (Server::handle_slave_request(MMDSSlaveRequest*)+0x8fc) [0x67019c]
8: (Server::dispatch(Message*)+0x69b) [0x670efb]
9: (MDSRank::handle_deferrable_message(Message*)+0x80c) [0x5f607c]
10: (MDSRank::_dispatch(Message*, bool)+0x1e1) [0x5ffd91]
11: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x600ee5]
12: (MDSDaemon::ms_dispatch(Message*)+0xc3) [0x5e8043]
13: (DispatchQueue::entry()+0x78b) [0xab7d0b]
14: (DispatchQueue::DispatchThread::entry()+0xd) [0x97b07d]
15: (()+0x8182) [0x7f89e6bec182]
16: (clone()+0x6d) [0x7f89e5ced47d]
1 jobs: ['327574']
</pre>
<p>From: <a class="external" href="http://pulpito.ceph.com/pdonnell-2016-07-21_13:20:27-multimds-master---basic-mira/327574/">http://pulpito.ceph.com/pdonnell-2016-07-21_13:20:27-multimds-master---basic-mira/327574/</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75269
2016-07-24T21:04:57Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Segmentation fault in another test which may be related:</p>
<pre>
Dead: 2016-07-19T01:26:57.608 INFO:tasks.workunit.client.0.mira064.stderr:/home/ubuntu/cephtest/workunit.client.0/suites/fsstress.sh: line 1: 3178 Terminated $command
ceph version v11.0.0-709-g12c0683 (12c068365c43a140fe1fe23bf68318342710e84d)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x17b61e7]
2: ceph-mds() [0x17b549f]
3: (()+0x10340) [0x7f65805d6340]
4: (MDSCacheObject::get(int)+0x13) [0x14454d5]
5: (MutationImpl::pin(MDSCacheObject*)+0x42) [0x14a3c8c]
6: (Server::handle_slave_rename_prep(std::shared_ptr<MDRequestImpl>&)+0x92c) [0x1434eb2]
7: (Server::dispatch_slave_request(std::shared_ptr<MDRequestImpl>&)+0xc33) [0x14034db]
8: (Server::handle_slave_request(MMDSSlaveRequest*)+0x1137) [0x140187f]
9: (Server::dispatch(Message*)+0x976) [0x13f1b84]
10: (MDSRank::handle_deferrable_message(Message*)+0xa71) [0x13a8057]
11: (MDSRank::_dispatch(Message*, bool)+0x3bc) [0x13a62c2]
12: (MDSRankDispatcher::ms_dispatch(Message*)+0x34) [0x13a5ef0]
13: (MDSDaemon::ms_dispatch(Message*)+0x21d) [0x13782bf]
14: (Messenger::ms_deliver_dispatch(Message*)+0x98) [0x1af71bc]
15: (DispatchQueue::entry()+0x5dd) [0x1af62d9]
16: (DispatchQueue::DispatchThread::entry()+0x1c) [0x1902046]
17: (Thread::entry_wrapper()+0xc1) [0x19ef733]
18: (Thread::_entry_func(void*)+0x18) [0x19ef668]
19: (()+0x8182) [0x7f65805ce182]
20: (clone()+0x6d) [0x7f657f6cf47d]
1 jobs: ['321707']
suites: ['clusters/3-mds.yaml', 'debug/mds_client.yaml', 'fs/btrfs.yaml', 'inline/no.yaml', 'mount/cfuse.yaml', 'multimds/basic/{ceph/base.yaml', 'overrides/whitelist_wrongly_marked_down.yaml', 'tasks/suites_fsstress.yaml}']
</pre>
<p><a class="external" href="http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/">http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75273
2016-07-25T02:59:26Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>please enable mds_debug</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75308
2016-07-25T14:13:50Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Zheng, which setting is that and how do I enable it? Sorry...</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75310
2016-07-25T14:16:29Z
John Spray
jcspray@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-5 priority-high3 closed" href="/issues/16807">Bug #16807</a>: Crash in handle_slave_rename_prep</i> added</li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75312
2016-07-25T14:16:45Z
John Spray
jcspray@gmail.com
<ul></ul><p>I've opened a separate ticket for the segfault, seems likely to be it's own issue (<a class="external" href="http://tracker.ceph.com/issues/16807">http://tracker.ceph.com/issues/16807</a>)</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75364
2016-07-26T13:55:28Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>please add a line "debug mds = 10" to ceph.conf</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75365
2016-07-26T14:02:02Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Zheng, I think we already have "debug mds = 20", right? From the config for this run: <a class="external" href="http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/">http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=75670
2016-08-01T02:36:33Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Here's another instance of the assertion failure on a more recent master branch:</p>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/pdonnell-2016-07-29_12:44:23-multimds-master---basic-mira/340811/teuthology.log">http://qa-proxy.ceph.com/teuthology/pdonnell-2016-07-29_12:44:23-multimds-master---basic-mira/340811/teuthology.log</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=79097
2016-09-30T03:26:42Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>Patrick Donnelly wrote:</p>
<blockquote>
<p>Zheng, I think we already have "debug mds = 20", right? From the config for this run: <a class="external" href="http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/">http://pulpito.ceph.com/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/</a></p>
</blockquote>
<p>the problem is that there is no log in <a class="external" href="http://qa-proxy.ceph.com/teuthology/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/">http://qa-proxy.ceph.com/teuthology/pdonnell-2016-07-18_20:02:54-multimds-master---basic-mira/321707/</a>. (I think all multimds runs do not have no log, which makes diagnose impossible)</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=79099
2016-09-30T03:33:48Z
Zheng Yan
ukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=80931
2016-11-09T13:51:27Z
John Spray
jcspray@gmail.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> set to <i>v12.0.0</i></li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84429
2017-01-20T12:19:54Z
John Spray
jcspray@gmail.com
<ul></ul><p>Another instance:<br /><a class="external" href="http://qa-proxy.ceph.com/teuthology/jspray-2017-01-19_17:59:40-multimds-wip-jcsp-testing-20170119b-testing-basic-smithi/731150/teuthology.log">http://qa-proxy.ceph.com/teuthology/jspray-2017-01-19_17:59:40-multimds-wip-jcsp-testing-20170119b-testing-basic-smithi/731150/teuthology.log</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84430
2017-01-20T12:27:13Z
John Spray
jcspray@gmail.com
<ul></ul><p>I noticed that failure while the test was still stuck trying to unmount the kernel client, so I went in and killed the ssh connection that was running umount so that it will (hopefully) proceed to gather the logs for us.</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84432
2017-01-20T12:28:53Z
John Spray
jcspray@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/8090">Bug #8090</a>: multimds: mds crash in check_rstats </i> added</li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84434
2017-01-20T12:29:00Z
John Spray
jcspray@gmail.com
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/8090">Bug #8090</a>: multimds: mds crash in check_rstats </i>)</li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84436
2017-01-20T12:29:10Z
John Spray
jcspray@gmail.com
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/8090">Bug #8090</a>: multimds: mds crash in check_rstats </i> added</li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84441
2017-01-20T14:24:50Z
John Spray
jcspray@gmail.com
<ul></ul><p>Hmm, well it didn't grab the logs for some reason but I did get the crashing MDS's log before the test tore down. It's in /home/jspray/16768 on teuthology.</p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=84589
2017-01-22T06:20:08Z
Zheng Yan
ukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/13052">https://github.com/ceph/ceph/pull/13052</a></p>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=86729
2017-02-24T14:10:59Z
Zheng Yan
ukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li></ul>
CephFS - Bug #16768: multimds: check_rstat assertion failure
https://tracker.ceph.com/issues/16768?journal_id=131364
2019-03-09T00:31:42Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Category</strong> deleted (<del><i>90</i></del>)</li><li><strong>Labels (FS)</strong> <i>multimds</i> added</li></ul>