https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-03-29T15:11:09ZCeph Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=683462016-03-29T15:11:09ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Tracker</strong> changed from <i>Bug</i> to <i>Support</i></li><li><strong>Category</strong> set to <i>fs/ceph</i></li><li><strong>Assignee</strong> set to <i>Zheng Yan</i></li></ul><p>What's the output of "ceph -s" during this time? Kinda looks like all your connections to the cluster are failed.</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=684482016-03-30T21:19:45ZRandy Orrrandy.orr@nimbix.net
<ul></ul><p>I believe I am hitting this issue as well:</p>
<p>Ubuntu 14.04<br />ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)</p>
<p>ceph -s reports healthy. Mapping and mounting an rbd works without issue, but mounting cephfs will consistently give the error.</p>
<p>I have tested the following kernels:</p>
<p>linux-image-3.19.0-42-generic - mounting cephfs works fine<br />linux-image-3.19.0-47-generic - mounting cephfs works fine<br />linux-image-3.19.0-49-generic - mounting cephfs fails with above error<br />linux-image-3.19.0-51-generic - mounting cephfs fails with above error<br />linux-image-3.19.0-56-generic - mounting cephfs fails with above error</p>
<p>I can reproduce this on multiple hosts in my environment. Is there any other information I can provide to help debug this issue?</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=684662016-03-31T03:53:37ZZheng Yanukernel@gmail.com
<ul></ul><p>ubuntu-vidid.git includes commit</p>
<pre>
commit ebeec2ef4cd85c971ab78a434a1f824a2fcb0447
Author: Arnd Bergmann <arnd@arndb.de>
Date: Wed Sep 30 15:04:42 2015 +0200
ceph: fix message length computation
BugLink: http://bugs.launchpad.net/bugs/1523652
commit 777d738a5e58ba3b6f3932ab1543ce93703f4873 upstream.
create_request_message() computes the maximum length of a message,
but uses the wrong type for the time stamp: sizeof(struct timespec)
may be 8 or 16 depending on the architecture, while sizeof(struct
ceph_timespec) is always 8, and that is what gets put into the
message.
Found while auditing the uses of timespec for y2038 problems.
Fixes: b8e69066d8af ("ceph: include time stamp in every MDS request")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Yan, Zheng <zyan@redhat.com>
Signed-off-by: Kamal Mostafa <kamal@canonical.com>
</pre>
<p>but does contains backport for commit</p>
<pre>
commit 1f041a89b4f22cf2e701514f4b8f73a8b1e06a3e
Author: Yan, Zheng <zyan@redhat.com>
Date: Tue Jan 13 15:20:52 2015 +0800
ceph: fix request time stamp encoding
struct timespec uses 'long' to present second and nanosecond. 'long'
is 64 bits on 64bits machine. ceph MDS expects time stamp to be
encoded as struct ceph_timespec, which uses 'u32' to present second
and nanosecond.
Signed-off-by: Yan, Zheng <zyan@redhat.com>
</pre> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=684672016-03-31T04:07:42ZZheng Yanukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>4</i></li></ul><p>sent bug report to <a class="email" href="mailto:kernel-team@lists.ubuntu.com">kernel-team@lists.ubuntu.com</a></p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=685712016-04-01T19:15:59ZKamal Mostafakamal@canonical.com
<ul></ul><p>The Ubuntu Kernel team is now tracking this bug here:<br /><a class="external" href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1564950">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1564950</a></p>
<p>And ...</p>
<p>I've constructed a test kernel for Ubuntu Vivid (amd64), comprised of 3.19.0-56.62 plus 1f041a89b4 (ceph: fix request time stamp encoding). Please confirm that this test kernel fixes the problem (installing just the linux-image-3.19...deb should be sufficient):</p>
<p><a class="external" href="http://kernel.ubuntu.com/~kamal/lp1564950/">http://kernel.ubuntu.com/~kamal/lp1564950/</a></p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=685802016-04-01T21:20:19ZRandy Orrrandy.orr@nimbix.net
<ul></ul><p>I can confirm that mounting a cephfs filesystem is successful using the provided test kernel.</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=685812016-04-01T21:22:52ZKamal Mostafakamal@canonical.com
<ul></ul><p>Thanks very much Randy. I'll see that the fix gets into the affected Ubuntu and -ckt stable kernels ASAP.</p>
<pre><code>-Kamal</code></pre> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=689382016-04-08T10:12:54ZDennis Kramer
<ul></ul><p>Thank you. <br />I can also confirm that it's fixed in the latest linux-generic-lts-vivid (3.19.0.58.41) which i installed from the default trusty repository.</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=689392016-04-08T10:54:06ZDennis Kramer
<ul></ul><p>Sorry, i was wrong. I'm still getting the same panic with 3.19.0.58.41</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=689632016-04-08T15:46:29ZKamal Mostafakamal@canonical.com
<ul></ul><p>This fix is still queued and pending in the Ubuntu repo's, but has not yet been released (as of 3.19.0-58.x).</p>
<p>In the meantime, here's an updated interim kernel for Ubuntu Vivid (amd64) for use on affected machines, comprised of 3.19.0-58 plus 1f041a89b4 (ceph: fix request time stamp encoding):</p>
<p><a class="external" href="http://kernel.ubuntu.com/~kamal/lp1564950-58/">http://kernel.ubuntu.com/~kamal/lp1564950-58/</a></p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=689982016-04-10T13:15:58ZIlya Dryomov
<ul></ul><p>v3.18.[26-30] are also affected. I've asked Sasha to queue up "ceph: fix request timestamp encoding".</p> Linux kernel client - Support #15302: Kernel panic when using CEPHFS (Hammer) /w kernel vivid-lts on Ubuntu 14.04https://tracker.ceph.com/issues/15302?journal_id=741732016-07-11T07:56:31ZZheng Yanukernel@gmail.com
<ul><li><strong>Status</strong> changed from <i>4</i> to <i>Resolved</i></li></ul>