https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2013-09-13T02:42:04ZCeph Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=274662013-09-13T02:42:04ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27466/diff?detail_id=29830">diff</a>)</li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=274672013-09-13T02:56:49ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27467/diff?detail_id=29831">diff</a>)</li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=274682013-09-13T03:01:57ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27468/diff?detail_id=29832">diff</a>)</li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=274702013-09-13T04:59:00ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27470/diff?detail_id=29834">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=274712013-09-13T05:32:22ZLoïc Dacharyloic@dachary.org
<ul><li><strong>File</strong> <a href="/attachments/download/1000/config.gz">config.gz</a> added</li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/27471/diff?detail_id=29837">diff</a>)</li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=276522013-09-17T09:45:51ZSage Weilsage@newdream.net
<ul></ul><p>It woudl probably be best ot send the xfs info to the <a class="email" href="mailto:xfs@oss.sgi.com">xfs@oss.sgi.com</a> list.</p>
<p>Also--you probably don't want to use 'nobarrier' unless you are <strong>very</strong> sure your raid card firmware is well behaved and nvram is reliable!</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=277682013-09-23T09:44:45ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Closed</i></li></ul> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=299802013-12-03T01:59:17ZLoïc Dacharyloic@dachary.org
<ul></ul><p>This really is a XFS problem that should be reported as follows : <a class="external" href="http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F">http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F</a></p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=299812013-12-03T02:01:47ZLoïc Dacharyloic@dachary.org
<ul></ul><p>Emmanuel Lacour ran into the same problem today and <a href="http://oss.sgi.com/pipermail/xfs/2013-December/032346.html" class="external">sent a bug report</a> with a <a href="http://people.easter-eggs.org/~manu/xfs.log" class="external">detailed description</a></p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=299822013-12-03T02:04:05ZEmmanuel Lacourelacour@easter-eggs.com
<ul></ul><p>I confirm, we had (several times since this cluster setup) a similar problem here. It seems related to some memory corruption on linux/xfs side which hang osd access to xfs fs. I just reported detail on xfs mailing list here:</p>
<p><a class="external" href="http://oss.sgi.com/pipermail/xfs/2013-December/032346.html">http://oss.sgi.com/pipermail/xfs/2013-December/032346.html</a></p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=300112013-12-04T00:21:39ZYann DupontYann.Dupont@univ-nantes.fr
<ul></ul><p>Loïc kindly generated this issue for me (lack of time, no open ID at the moment of the report... etc)</p>
<p>So it's obviously the same bug that the one reported by emmanuel on the XFS list (thanks emmanuel for taking time to do that) and it seems the problem is caused by 64k directories on XFS.</p>
<p>Dave chinner proposed a patch, under test on XFS folks side, but this one only covers one aspect of the problem.</p>
<p>See <a class="external" href="http://oss.sgi.com/archives/xfs/2013-12/msg00087.html">http://oss.sgi.com/archives/xfs/2013-12/msg00087.html</a></p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=300122013-12-04T00:23:21ZYann DupontYann.Dupont@univ-nantes.fr
<ul></ul><p>maybe the status should be changed from closed to something more appropriate, even if ceph isn't the culprit here ...</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=329262014-03-06T06:24:18ZYann DupontYann.Dupont@univ-nantes.fr
<ul></ul><p>One fix has been integrated in kernel commit b3f03bac8132207a20286d5602eda64500c19724<br />(3.14rc1).</p>
<p>No backport AFAICS.</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=420592014-09-29T16:49:32ZMark Nelsonmark.a.nelson@gmail.com
<ul></ul><p>fwiw, after upgrading the performance test nodes from Ubuntu 13.10 to Fedora Core 20, I appear to be hitting this under heavy load as well. confirmed that neither fedora 20 nor the RHEL7 kernel source contain b3f03ba.</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=461762015-01-06T17:45:30Zwei litxdyjsyz@gmail.com
<ul></ul><p>We face to this problem in centos 7, kernel 3.10.0-123.el7.x86_64<br />We download the kernel source <a class="external" href="http://vault.centos.org/7.0.1406/os/Source/SPackages/kernel-3.10.0-123.el7.src.rpm">http://vault.centos.org/7.0.1406/os/Source/SPackages/kernel-3.10.0-123.el7.src.rpm</a><br />Try to manual patch the kernel commit b3f03bac8132207a20286d5602eda64500c19724, but we found this commit already in kernel-3.10.0-123.el7.<br />It seems the redhat already patch this, we fire an issue in centos <a class="external" href="http://bugs.centos.org/view.php?id=8053">http://bugs.centos.org/view.php?id=8053</a><br />We find the smilar issue in kernel <a class="external" href="https://bugzilla.kernel.org/show_bug.cgi?id=73831">https://bugzilla.kernel.org/show_bug.cgi?id=73831</a></p>
<p>In our test env, the rbd order set to 27, block size is 128M. Is it possible relat to this? Or another bug in xfs.<br />When xfs happen to memory allocation deadlock, the osd will hang, the front side clinet IO will hang too.<br />Restart osd do not fix the problem, need restart the machine.</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=461782015-01-06T18:53:18Zwei litxdyjsyz@gmail.com
<ul></ul><p>Some ceph error log when the xfs memory allocation deadlock<br /><pre>
2015-01-05 16:29:14.011438 7fc7a4274700 -1 os/FileStore.cc: In function 'virtual int FileStore::read(coll_t, const ghobject_t&, uint64_t, size_t, ceph::bufferlist&, bool)' thread 7fc7a4274700 time 2015-01-05 16:29:13.913484
os/FileStore.cc: 2641: FAILED assert(allow_eio || !m_filestore_fail_eio || got != -5)
ceph version ()
1: (FileStore::read(coll_t, ghobject_t const&, unsigned long, unsigned long, ceph::buffer::list&, bool)+0xc0a) [0x891b0a]
2: (ReplicatedBackend::build_push_op(ObjectRecoveryInfo const&, ObjectRecoveryProgress const&, ObjectRecoveryProgress*, PushOp*, object_stat_sum_t*)+0x5e0) [0x7d4c90]
3: (ReplicatedBackend::handle_push_reply(pg_shard_t, PushReplyOp&, PushOp*)+0x266) [0x817456]
4: (ReplicatedBackend::do_push_reply(std::tr1::shared_ptr<OpRequest>)+0x13d) [0x817a5d]
5: (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x266) [0x922556]
6: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x303) [0x7afe73]
7: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a5) [0x6003f5]
8: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x203) [0x61b6a3]
9: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0xac) [0x65f70c]
10: (ThreadPool::worker(ThreadPool::WorkThread*)+0xb10) [0xa77cb0]
11: (ThreadPool::WorkThread::entry()+0x10) [0xa78ba0]
12: (()+0x7df3) [0x7fc7bd6e8df3]
13: (clone()+0x6d) [0x7fc7bc3d33dd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre></p>
</pre><br /> ceph version ()<br /> 1: /usr/bin/ceph-osd() [0x99d296]<br /> 2: (()+0xf130) [0x7fc7bd6f0130]<br /> 3: (gsignal()+0x39) [0x7fc7bc312989]<br /> 4: (abort()+0x148) [0x7fc7bc314098]<br /> 5: (_<em>gnu_cxx::</em>_verbose_terminate_handler()+0x165) [0x7fc7bcc169d5]<br /> 6: (()+0x5e946) [0x7fc7bcc14946]<br /> 7: (()+0x5e973) [0x7fc7bcc14973]<br /> 8: (()+0x5eb9f) [0x7fc7bcc14b9f]<br /> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ef) [0xa8725f]<br /> 10: (FileStore::read(coll_t, ghobject_t const&, unsigned long, unsigned long, ceph::buffer::list&, bool)+0xc0a) [0x891b0a]<br /> 11: (ReplicatedBackend::build_push_op(ObjectRecoveryInfo const&, ObjectRecoveryProgress const&, ObjectRecoveryProgress*, PushOp*, object_stat_sum_t*)+0x5e0) [0x7d4c90]<br /> 12: (ReplicatedBackend::handle_push_reply(pg_shard_t, PushReplyOp&, PushOp*)+0x266) [0x817456]<br /> 13: (ReplicatedBackend::do_push_reply(std::tr1::shared_ptr<OpRequest>)+0x13d) [0x817a5d]<br /> 14: (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x266) [0x922556]<br /> 15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x303) [0x7afe73]<br /> 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a5) [0x6003f5]<br /> 17: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x203) [0x61b6a3]<br /> 18: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0xac) [0x65f70c]<br /> 19: (ThreadPool::worker(ThreadPool::WorkThread*)+0xb10) [0xa77cb0]<br /> 20: (ThreadPool::WorkThread::entry()+0x10) [0xa78ba0]<br /> 21: (()+0x7df3) [0x7fc7bd6e8df3]<br /> 22: (clone()+0x6d) [0x7fc7bc3d33dd]<br /> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-01-05 17:00:03.024481 7f4a6ed837c0 0 ceph version (), process ceph-osd, pid 5117<br />2015-01-05 17:00:03.043106 7f4a6ed837c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount detected xfs (libxfs)<br />2015-01-05 17:00:03.043115 7f4a6ed837c0 1 filestore(/var/lib/ceph/osd/ceph-9) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs<br />2015-01-05 17:00:03.044863 7f4a6ed837c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is supported and appears to work<br />2015-01-05 17:00:03.044873 7f4a6ed837c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option<br />2015-01-05 17:00:03.045124 7f4a6ed837c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)<br />2015-01-05 17:00:03.045170 7f4a6ed837c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) detect_feature: extsize is disabled by conf<br />2015-01-05 17:00:03.150770 7f4a6ed837c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled<br />2015-01-05 17:00:40.302339 7f4a6ed837c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway<br />2015-01-05 17:00:40.302376 7f4a6ed837c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 20: 4294967296 bytes, block size 4096 bytes, directio = 1, aio = 0<br />2015-01-05 17:00:41.104992 7f4a6ed837c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 20: 4294967296 bytes, block size 4096 bytes, directio = 1, aio = 0<br />2015-01-05 17:00:41.123568 7f4a6ed837c0 1 journal close /var/lib/ceph/osd/ceph-9/journal<br /></pre> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=465002015-01-15T08:10:54ZYann DupontYann.Dupont@univ-nantes.fr
<ul></ul><p>Is this with 64k directories ?</p>
<p>We don't have the problem anymore - all our OSD now use standard xfs format - (that is, no 64k directory).</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=473712015-02-03T08:34:30Zwei litxdyjsyz@gmail.com
<ul></ul><p>@Yann Dupont<br />Yes, I mkfs args is:<br /><pre>
mkfs.xfs -f -d agcount=32 -l size=1024m -n size=64k /dev/sdb1
</pre><br />How much do you prefer for this?</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=473722015-02-03T09:11:00ZYann DupontYann.Dupont@univ-nantes.fr
<ul></ul><p>Well, in fact we were plagued by this issue for a long period last year. It was very sporadic. I'm not sure -n size=64k was a good idea anyway.</p>
<p>So, after a while, we decided to reformat all our nodes, one by one, and stick with XFS default parameters (that is, plain simple mkfs.xfs without options). That was cumbersome to do (we had to convert all our OSD), but in the end, that was a wise move : no problems since.</p>
<p>I hope you don't have much OSD to convert.</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=474322015-02-04T01:42:39Zwei litxdyjsyz@gmail.com
<ul></ul><p>@Yann Dupont<br />Thank you very much! I will do so.</p> Ceph - Bug #6301: ceph-osd hung by XFS using linux 3.10https://tracker.ceph.com/issues/6301?journal_id=480322015-02-12T23:16:40ZWarren Wangwarren@wangspeed.com
<ul></ul><p>I know this isn't strictly a Ceph problem, but hopefully I can save some folks some grief. I just spent 3 weeks looking at this problem. For Ubuntu users, kernel 3.13.0-40 and above include the fix for the xfs contiguous allocation bug.</p>