https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-05-14T14:41:44ZCeph CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1131332018-05-14T14:41:44ZJohn Sprayjcspray@gmail.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>CephFS</i></li></ul><p>Any chance you can reproduce this with debuginfo packages installed, so that we can get meaningful backtraces?</p> CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1131872018-05-14T22:04:21ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> changed from <i>v12.2.5</i> to <i>v14.0.0</i></li><li><strong>Source</strong> set to <i>Community (user)</i></li><li><strong>Tags</strong> deleted (<del><i>mds, cephfs, crash,</i></del>)</li><li><strong>Affected Versions</strong> deleted (<del><i>v12.2.4, v12.2.5</i></del>)</li><li><strong>ceph-qa-suite</strong> deleted (<del><i>fs</i></del>)</li><li><strong>Component(FS)</strong> <i>MDS</i> added</li><li><strong>Labels (FS)</strong> <i>crash</i> added</li></ul> CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1137832018-05-22T20:12:47ZSean Sullivanlookcrabs@gmail.com
<ul></ul><p>John Spray wrote:</p>
<blockquote>
<p>Any chance you can reproduce this with debuginfo packages installed, so that we can get meaningful backtraces?</p>
</blockquote>
<p>Hopefully this helps. I'm a dummy and not exactly sure how to do this well. I also missed this reply, sorry again. I have all the packages ceph-*-dbg installed but this time I attached to ceph-mds with gdb prior to the crash:</p>
<p><a class="external" href="https://pastebin.com/kw4bZVZT">https://pastebin.com/kw4bZVZT</a> -- kh09-9<br /><a class="external" href="https://pastebin.com/sYZQx0ER">https://pastebin.com/sYZQx0ER</a> -- kh10-9</p>
<p>-----------------------------------------------<br />List of dbg packages installed on one of the mds servers (same installed on both):<br />root@kh09-8:~# dpkg -l | grep -i dbg<br />ii ceph-base-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-base<br />ii ceph-common-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-common<br />ii ceph-fuse-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-fuse<br />ii ceph-mds-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-mds<br />ii ceph-mgr-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-mgr<br />ii ceph-mon-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-mon<br />ii ceph-osd-dbg 12.2.5-1xenial amd64 debugging symbols for ceph-osd<br />ii libc6-dbg:amd64 2.23-0ubuntu10 amd64 GNU C Library: detached debugging symbols<br />ii libcephfs2-dbg 12.2.5-1xenial amd64 debugging symbols for libcephfs2<br />ii librados2-dbg 12.2.5-1xenial amd64 debugging symbols for librados<br />ii librbd1-dbg 12.2.5-1xenial amd64 debugging symbols for librbd1<br />ii librgw2-dbg 12.2.5-1xenial amd64 debugging symbols for librbd1<br />ii radosgw-dbg 12.2.5-1xenial amd64 debugging symbols for radosgw<br />ii rbd-fuse-dbg 12.2.5-1xenial amd64 debugging symbols for rbd-fuse<br />ii rbd-mirror-dbg 12.2.5-1xenial amd64 debugging symbols for rbd-mirror<br />ii rbd-nbd-dbg 12.2.5-1xenial amd64 debugging symbols for rbd-nbd</p>
<p>so I'm not sure why the symbols are not loaded in the original traces. I hope these new traces help.</p> CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1139282018-05-24T02:21:19ZZheng Yanukernel@gmail.com
<ul></ul><p>The crash was at "mdr->tracedn = mdr->dn[ 0].back()", because mdr->dn[ 0] is empty. request that triggered the crash is something like "lookup #0x1//"</p>
<pre>
dout(10) << "reply to stat on " << *req << dendl;
mdr->tracei = ref;
if (is_lookup)
mdr->tracedn = mdr->dn[0].back();
respond_to_request(mdr, 0);
</pre>
<p>Following patch can prevents kclient from sending malformed lookup request. But the real bug should be located in aufs, it should never revalidate root dentry.<br /><pre>
diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c
index f1d9c6cc0491..3c2b1b553654 100644
--- a/fs/ceph/dir.c
+++ b/fs/ceph/dir.c
@@ -1197,6 +1197,9 @@ static int ceph_d_revalidate(struct dentry *dentry, unsigned int flags)
struct dentry *parent;
struct inode *dir;
+ if (IS_ROOT(dentry))
+ return 1;
+
if (flags & LOOKUP_RCU) {
parent = READ_ONCE(dentry->d_parent);
dir = d_inode_rcu(parent);
</pre></p> CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1309752019-03-07T23:21:34ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> changed from <i>v14.0.0</i> to <i>v15.0.0</i></li></ul> CephFS - Bug #23972: Ceph MDS Crash from client mounting aufs over cephfshttps://tracker.ceph.com/issues/23972?journal_id=1310752019-03-07T23:32:22ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Target version</strong> deleted (<del><i>v15.0.0</i></del>)</li></ul>