https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2020-05-13T19:26:15Z
Ceph
Ceph - Bug #45537: librados calls after fork() result in lockdep aborts
https://tracker.ceph.com/issues/45537?journal_id=165829
2020-05-13T19:26:15Z
David Disseldorp
ddiss@suse.de
<ul><li><strong>File</strong> <a href="/attachments/download/4873/0001-test-librados-fork-test.patch">0001-test-librados-fork-test.patch</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/4873/0001-test-librados-fork-test.patch">View</a> added</li></ul><p>The attached unit test atop current master (3252b19084bf7b6864733b4183eb8ef4d937d691) results in the following abort from the ms_local thread:</p>
<p><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:2178 (Closed)" href="https://tracker.ceph.com/issues/2">#2</a> 0x00007fffee266739 in ceph::__ceph_abort (file=0x7fffeeba1398 "/home/david/zram/work/ceph/src/common/lockdep.cc", line=336, <br /> func=0x7fffeeba3080 <lockdep_will_lock(char const*, int, bool, bool)::__PRETTY_FUNCTION__> "int lockdep_will_lock(const char*, int, bool, bool)", msg=...)<br /> at /home/david/zram/work/ceph/src/common/assert.cc:198<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: leaked dentry ref on umount (Closed)" href="https://tracker.ceph.com/issues/3">#3</a> 0x00007fffee35e348 in lockdep_will_lock (name=0x555555ce1960 "Messenger::DispatchQueue::local_delivery_lockradosclient", id=20, force_backtrace=false, recursive=false)<br /> at /home/david/zram/work/ceph/src/common/lockdep.cc:336<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: lockdep warning in socket code (Closed)" href="https://tracker.ceph.com/issues/4">#4</a> 0x00007fffee369d6a in ceph::mutex_debug_detail::mutex_debugging_base::_will_lock (this=0x555555cddc00, recursive=false)<br /> at /home/david/zram/work/ceph/src/common/mutex_debug.cc:49<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: ./rados lspools sometimes hangs after listing all pools? (Closed)" href="https://tracker.ceph.com/issues/5">#5</a> 0x00007fffee105882 in ceph::mutex_debug_detail::mutex_debug_impl<false>::lock (this=0x555555cddc00, no_lockdep=false)<br /> at /home/david/zram/work/ceph/src/common/mutex_debug.h:180<br /><a class="issue tracker-2 status-6 priority-3 priority-lowest closed" title="Feature: libceph could use a backward-compatible-to function (Rejected)" href="https://tracker.ceph.com/issues/6">#6</a> 0x00007fffee105980 in std::unique_lock<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock (this=0x7fffe94930b0) at /usr/include/c++/7/bits/std_mutex.h:267<br /><a class="issue tracker-6 status-3 priority-3 priority-lowest closed" title="Documentation: Document Monitor Commands (Resolved)" href="https://tracker.ceph.com/issues/7">#7</a> 0x00007fffee103856 in std::unique_lock<ceph::mutex_debug_detail::mutex_debug_impl<false> >::unique_lock (this=0x7fffe94930b0, __m=...)<br /> at /usr/include/c++/7/bits/std_mutex.h:197<br /><a class="issue tracker-3 status-5 priority-4 priority-default closed" title="Support: Document differences from S3 (Closed)" href="https://tracker.ceph.com/issues/8">#8</a> 0x00007fffee43cbba in DispatchQueue::run_local_delivery (this=0x555555cdda48) at /home/david/zram/work/ceph/src/msg/DispatchQueue.cc:114<br /><a class="issue tracker-2 status-8 priority-3 priority-lowest closed" title="Feature: Access unimported data (Won't Fix)" href="https://tracker.ceph.com/issues/9">#9</a> 0x00007fffee59b436 in DispatchQueue::LocalDeliveryThread::entry (this=0x555555cddcf0) at /home/david/zram/work/ceph/src/msg/DispatchQueue.h:115<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: osd: Replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/10">#10</a> 0x00007fffee201ed6 in Thread::entry_wrapper (this=0x555555cddcf0) at /home/david/zram/work/ceph/src/common/Thread.cc:84<br /><a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="Cleanup: mds: replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/11">#11</a> 0x00007fffee201e54 in Thread::_entry_func (arg=0x555555cddcf0) at /home/david/zram/work/ceph/src/common/Thread.cc:71<br /><a class="issue tracker-2 status-3 priority-3 priority-lowest closed" title="Feature: uclient: Make cap handling smarter (Resolved)" href="https://tracker.ceph.com/issues/12">#12</a> 0x00007ffff776c4f9 in start_thread () from /lib64/libpthread.so.0<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: uclient: Make readdir use the cache (Resolved)" href="https://tracker.ceph.com/issues/13">#13</a> 0x00007fffeb0b6f2f in clone () from /lib64/libc.so.6</p>
<p>While the main thread is calling _rados_connect():</p>
<p>(gdb) thread 2.1<br />[Switching to thread 2.1 (Thread 0x7ffff7fd84c0 (LWP 86009))]<br />#0 0x00007ffff7775c3d in __lll_lock_wait () from /lib64/libpthread.so.0<br />(gdb) bt <br />#0 0x00007ffff7775c3d in __lll_lock_wait () from /lib64/libpthread.so.0<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a> 0x00007ffff776ee95 in pthread_mutex_lock () from /lib64/libpthread.so.0<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:2178 (Closed)" href="https://tracker.ceph.com/issues/2">#2</a> 0x00007fffee35e9ed in lockdep_will_unlock (name=0x555555bbfaf0 "AsyncMessenger::lock", id=22) at /home/david/zram/work/ceph/src/common/lockdep.cc:383<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: leaked dentry ref on umount (Closed)" href="https://tracker.ceph.com/issues/3">#3</a> 0x00007fffee369df8 in ceph::mutex_debug_detail::mutex_debugging_base::_will_unlock (this=0x555555cddde0) at /home/david/zram/work/ceph/src/common/mutex_debug.cc:55<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: lockdep warning in socket code (Closed)" href="https://tracker.ceph.com/issues/4">#4</a> 0x00007fffee105917 in ceph::mutex_debug_detail::mutex_debug_impl<false>::unlock (this=0x555555cddde0, no_lockdep=false)<br /> at /home/david/zram/work/ceph/src/common/mutex_debug.h:194<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: ./rados lspools sometimes hangs after listing all pools? (Closed)" href="https://tracker.ceph.com/issues/5">#5</a> 0x00007fffee1246e8 in std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::~lock_guard (this=0x7fffffffd270, __in_chrg=<optimized out>)<br /> at /usr/include/c++/7/bits/std_mutex.h:168<br /><a class="issue tracker-2 status-6 priority-3 priority-lowest closed" title="Feature: libceph could use a backward-compatible-to function (Rejected)" href="https://tracker.ceph.com/issues/6">#6</a> 0x00007fffee592e28 in AsyncMessenger::ready (this=0x555555cdd6b0) at /home/david/zram/work/ceph/src/msg/async/AsyncMessenger.cc:334<br /><a class="issue tracker-6 status-3 priority-3 priority-lowest closed" title="Documentation: Document Monitor Commands (Resolved)" href="https://tracker.ceph.com/issues/7">#7</a> 0x00007ffff7a4a461 in Messenger::add_dispatcher_head (this=0x555555cdd6b0, d=0x555555cd6080) at /home/david/zram/work/ceph/src/msg/Messenger.h:390<br /><a class="issue tracker-3 status-5 priority-4 priority-default closed" title="Support: Document differences from S3 (Closed)" href="https://tracker.ceph.com/issues/8">#8</a> 0x00007ffff7a41778 in librados::v14_2_0::RadosClient::connect (this=0x555555cd5840) at /home/david/zram/work/ceph/src/librados/RadosClient.cc:275<br /><a class="issue tracker-2 status-8 priority-3 priority-lowest closed" title="Feature: Access unimported data (Won't Fix)" href="https://tracker.ceph.com/issues/9">#9</a> 0x00007ffff79b871e in _rados_connect (cluster=0x555555cd5840) at /home/david/zram/work/ceph/src/librados/librados_c.cc:178<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: osd: Replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/10">#10</a> 0x000055555566463c in forker_start_proc (cluster_parent=0x555555bb43a0) at /home/david/zram/work/ceph/src/test/librados/misc.cc:383<br /><a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="Cleanup: mds: replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/11">#11</a> 0x0000555555664c8e in LibRadosMisc_Forker_Test::TestBody (this=0x555555aa09b0) at /home/david/zram/work/ceph/src/test/librados/misc.cc:421<br /><a class="issue tracker-2 status-3 priority-3 priority-lowest closed" title="Feature: uclient: Make cap handling smarter (Resolved)" href="https://tracker.ceph.com/issues/12">#12</a> 0x00005555556e7f01 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x555555aa09b0, method=&virtual testing::Test::TestBody(), <br /> location=0x555555710ffb "the test body") at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2593<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: uclient: Make readdir use the cache (Resolved)" href="https://tracker.ceph.com/issues/13">#13</a> 0x00005555556e0c59 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x555555aa09b0, method=&virtual testing::Test::TestBody(), <br /> location=0x555555710ffb "the test body") at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2629<br /><a class="issue tracker-1 status-10 priority-4 priority-default closed" title="Bug: osd: pg split breaks if not all osds are up (Duplicate)" href="https://tracker.ceph.com/issues/14">#14</a> 0x00005555556bcb10 in testing::Test::Run (this=0x555555aa09b0) at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2668<br /><a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: mds rejoin: invented dirfrags (MDCache.cc:3469) (Resolved)" href="https://tracker.ceph.com/issues/15">#15</a> 0x00005555556bd4a7 in testing::TestInfo::Run (this=0x5555559a55e0) at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2845<br /><a class="issue tracker-1 status-3 priority-5 priority-high3 closed" title="Bug: mds restart vs dbench (Resolved)" href="https://tracker.ceph.com/issues/16">#16</a> 0x00005555556bdbba in testing::TestSuite::Run (this=0x5555559e4940) at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2977<br /><a class="issue tracker-1 status-6 priority-4 priority-default closed" title="Bug: rm -r failure (Rejected)" href="https://tracker.ceph.com/issues/17">#17</a> 0x00005555556c97ea in testing::internal::UnitTestImpl::RunAllTests (this=0x5555559e7c30) at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:5518<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: reconnect fixups (Resolved)" href="https://tracker.ceph.com/issues/18">#18</a> 0x00005555556e9121 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555559e7c30, <br /> method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555556c93d0 <testing::internal::UnitTestImpl::RunAllTests()>, <br /> location=0x5555557119e8 "auxiliary test code (environments or event listeners)") at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2593<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: rbd (Resolved)" href="https://tracker.ceph.com/issues/19">#19</a> 0x00005555556e1cfc in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x5555559e7c30, <br /> method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x5555556c93d0 <testing::internal::UnitTestImpl::RunAllTests()>, <br /> location=0x5555557119e8 "auxiliary test code (environments or event listeners)") at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:2629<br /><a class="issue tracker-2 status-3 priority-5 priority-high3 closed" title="Feature: client: recover from a killed session (w/ blacklist) (Resolved)" href="https://tracker.ceph.com/issues/20">#20</a> 0x00005555556c8049 in testing::UnitTest::Run (this=0x555555962a80 <testing::UnitTest::GetInstance()::instance>)<br /> at /home/david/zram/work/ceph/src/googletest/googletest/src/gtest.cc:5105<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: optionally use libatomic for atomic_t (Resolved)" href="https://tracker.ceph.com/issues/21">#21</a> 0x00005555556986c2 in RUN_ALL_TESTS () at /home/david/zram/work/ceph/src/googletest/googletest/include/gtest/gtest.h:2472<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:253 (Closed)" href="https://tracker.ceph.com/issues/22">#22</a> 0x0000555555698577 in main (argc=1, argv=0x7fffffffe408) at /home/david/zram/work/ceph/src/test/unit.cc:45</p>
Ceph - Bug #45537: librados calls after fork() result in lockdep aborts
https://tracker.ceph.com/issues/45537?journal_id=194537
2021-05-09T17:22:53Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Won't Fix</i></li></ul><p>librados has network state; you will need to play nicely around forking it to make sure things continue to work.</p>