Project

General

Profile

Actions

Bug #20506

closed

OSD memory leak (possibly Bluestore-related?) in rgw suite in kraken 11.2.1 integration testing

Added by Nathan Cutler almost 7 years ago. Updated over 6 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

<error>
  <unique>0x85</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>246,694 (96 direct, 246,598 indirect) bytes in 1 blocks are definitely lost in loss record 134 of 136</text>
    <leakedbytes>246694</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA10C8A8</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>423</line>
    </frame>
    <frame>
      <ip>0x96F467</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>allocate</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/include</dir>
      <file>mempool.h</file>
      <line>298</line>
    </frame>
    <frame>
      <ip>0x96F467</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>std::__detail::_Hash_node&lt;std::pair&lt;ghobject_t const, boost::intrusive_ptr&lt;BlueStore::Onode&gt; &gt;, true&gt;* std::_Hashtable&lt;ghobject_t, std::pair&lt;ghobject_t const, boost::intrusive_ptr&lt;BlueStore::Onode&gt; &gt;, mempool::pool_allocator&lt;(mempool::pool_index_t)6, std::pair&lt;ghobject_t, boost::intrusive_ptr&lt;BlueStore::Onode&gt; &gt; &gt;, std::__detail::_Select1st, std::equal_to&lt;ghobject_t&gt;, std::hash&lt;ghobject_t&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;true, false, true&gt; &gt;::_M_allocate_node&lt;std::piecewise_construct_t const&amp;, std::tuple&lt;ghobject_t const&amp;&gt;, std::tuple&lt;&gt; &gt;(std::piecewise_construct_t const&amp;, std::tuple&lt;ghobject_t const&amp;&gt;&amp;&amp;, std::tuple&lt;&gt;&amp;&amp;)</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>hashtable.h</file>
      <line>724</line>
    </frame>
    <frame>
      <ip>0x97221D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>std::__detail::_Map_base&lt;ghobject_t, std::pair&lt;ghobject_t const, boost::intrusive_ptr&lt;BlueStore::Onode&gt; &gt;, mempool::pool_allocator&lt;(mempool::pool_index_t)6, std::pair&lt;ghobject_t, boost::intrusive_ptr&lt;BlueStore::Onode&gt; &gt; &gt;, std::__detail::_Select1st, std::equal_to&lt;ghobject_t&gt;, std::hash&lt;ghobject_t&gt;, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits&lt;true, false, true&gt;, true&gt;::operator[](ghobject_t const&amp;)</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>hashtable_policy.h</file>
      <line>493</line>
    </frame>
    <frame>
      <ip>0x934576</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>operator[]</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>unordered_map.h</file>
      <line>596</line>
    </frame>
    <frame>
      <ip>0x934576</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::OnodeSpace::add(ghobject_t const&amp;, boost::intrusive_ptr&lt;BlueStore::Onode&gt;)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>1190</line>
    </frame>
    <frame>
      <ip>0x935204</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::Collection::get_onode(ghobject_t const&amp;, bool)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>2462</line>
    </frame>
    <frame>
      <ip>0x961B43</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>7121</line>
    </frame>
    <frame>
      <ip>0x9632B8</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector&lt;ObjectStore::Transaction, std::allocator&lt;ObjectStore::Transaction&gt; &gt;&amp;, std::shared_ptr&lt;TrackedOp&gt;, ThreadPool::TPHandle*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>6961</line>
    </frame>
    <frame>
      <ip>0x5AD5E4</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>queue_transactions</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os</dir>
      <file>ObjectStore.h</file>
      <line>1453</line>
    </frame>
    <frame>
      <ip>0x5AD5E4</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&amp;&amp;, Context*, Context*, Context*, std::shared_ptr&lt;TrackedOp&gt;, ThreadPool::TPHandle*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/os</dir>
      <file>ObjectStore.h</file>
      <line>1441</line>
    </frame>
    <frame>
      <ip>0x56C798</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::handle_osd_map(MOSDMap*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/osd</dir>
      <file>OSD.cc</file>
      <line>7012</line>
    </frame>
    <frame>
      <ip>0x57F820</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::_dispatch(Message*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/osd</dir>
      <file>OSD.cc</file>
      <line>6443</line>
    </frame>
    <frame>
      <ip>0x57FECD</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::ms_dispatch(Message*)</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/osd</dir>
      <file>OSD.cc</file>
      <line>5991</line>
    </frame>
    <frame>
      <ip>0xD88F11</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ms_deliver_dispatch</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/msg</dir>
      <file>Messenger.h</file>
      <line>593</line>
    </frame>
    <frame>
      <ip>0xD88F11</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>DispatchQueue::entry()</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/msg</dir>
      <file>DispatchQueue.cc</file>
      <line>197</line>
    </frame>
    <frame>
      <ip>0xC55E4C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>DispatchQueue::DispatchThread::entry()</fn>
      <dir>/usr/src/debug/ceph-11.2.0-605-g5fbe1d9/src/msg</dir>
      <file>DispatchQueue.h</file>
      <line>102</line>
    </frame>
    <frame>
      <ip>0xC22DDC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD38076C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

/a/smithfarm-2017-07-05_06:44:30-rgw-wip-kraken-backports-distro-basic-smithi/1363474/remote/smithi023/log/valgrind/osd.2.log.gz

Actions #1

Updated by Nathan Cutler almost 7 years ago

  • Subject changed from Bluestore-related OSD memory leak triggered by rgw suite in kraken 11.2.1 integration testing to OSD memory leak (possibly Bluestore-related?) in rgw suite in kraken 11.2.1 integration testing
Actions #2

Updated by Sage Weil over 6 years ago

  • Project changed from Ceph to bluestore
Actions #3

Updated by Sage Weil over 6 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF