Project

General

Profile

Actions
Copy link

Bug #17417

closed

/var/log/ceph/valgrind/osd.1.log: <kind>Leak_DefinitelyLost</kind>

Added by David Zafman over 7 years ago. Updated almost 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/dzafman-2016-09-22_17:11:48-rados-wip-zafman-testing-distro-basic-smithi/431916

Actions
Copy link
 #1

Updated by Kefu Chai over 7 years ago

the log is gone, this is a new occurrence.

<error>
  <unique>0x55</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,041 (32 direct, 1,009 indirect) bytes in 1 blocks are definitely lost in loss record 86 of 94</text>
    <leakedbytes>1041</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA0EB203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/ext</dir>
      <file>new_allocator.h</file>
      <line>104</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>502</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>1561</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>1021</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>push_back</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/include</dir>
      <file>buffer.h</file>
      <line>752</line>
    </frame>
    <frame>
      <ip>0xD77E32</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>2077</line>
    </frame>
    <frame>
      <ip>0xD84C00</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1687</line>
    </frame>
    <frame>
      <ip>0xD8DDAC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>51</line>
    </frame>
    <frame>
      <ip>0xC1FEDC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD35173C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

and

<error>
  <unique>0x56</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,064 (56 direct, 1,008 indirect) bytes in 1 blocks are definitely lost in loss record 87 of 94</text>
    <leakedbytes>1064</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA0EB203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0xD4118C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/ext</dir>
      <file>new_allocator.h</file>
      <line>104</line>
    </frame>
    <frame>
      <ip>0xD4118C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>_M_get_node</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_tree.h</file>
      <line>370</line>
    </frame>
    <frame>
      <ip>0xD4118C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>_M_create_node&lt;const std::piecewise_construct_t&amp;, std::tuple&lt;const std::pair&lt;long unsigned int, long unsigned int&gt;&amp;&gt;, std::tuple&lt;&gt; &gt;</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_tree.h</file>
      <line>403</line>
    </frame>
    <frame>
      <ip>0xD4118C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>std::_Rb_tree_iterator&lt;std::pair&lt;std::pair&lt;unsigned long, unsigned long&gt; const, std::pair&lt;unsigned int, unsigned int&gt; &gt; &gt; std::_Rb_tree&lt;std::pair&lt;unsigned long, unsigned long&gt;, std::pair&lt;std::pair&lt;unsigned long, unsigned long&gt; const, std::pair&lt;unsigned int, unsigned int&gt; &gt;, std::_Select1st&lt;std::pair&lt;std::pair&lt;unsigned long, unsigned long&gt; const, std::pair&lt;unsigned int, unsigned int&gt; &gt; &gt;, std::less&lt;std::pair&lt;unsigned long, unsigned long&gt; &gt;, std::allocator&lt;std::pair&lt;std::pair&lt;unsigned long, unsigned long&gt; const, std::pair&lt;unsigned int, unsigned int&gt; &gt; &gt; &gt;::_M_emplace_hint_unique&lt;std::piecewise_construct_t const&amp;, std::tuple&lt;std::pair&lt;unsigned long, unsigned long&gt; const&amp;&gt;, std::tuple&lt;&gt; &gt;(std::_Rb_tree_const_iterator&lt;std::pair&lt;std::pair&lt;unsigned long, unsigned long&gt; const, std::pair&lt;unsigned int, unsigned int&gt; &gt; &gt;, std::piecewise_construct_t const&amp;, std::tuple&lt;std::pair&lt;unsigned long, unsigned long&gt; const&amp;&gt;&amp;&amp;, std::tuple&lt;&gt;&amp;&amp;)</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_tree.h</file>
      <line>1669</line>
    </frame>
    <frame>
      <ip>0xD3D1C0</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>set_crc</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/common</dir>
      <file>buffer.cc</file>
      <line>229</line>
    </frame>
    <frame>
      <ip>0xD3D1C0</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ceph::buffer::list::crc32c(unsigned int) const</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/common</dir>
      <file>buffer.cc</file>
      <line>2411</line>
    </frame>
    <frame>
      <ip>0xBF0011</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>decode_message(CephContext*, int, ceph_msg_header&amp;, ceph_msg_footer&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;)</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg</dir>
      <file>Message.cc</file>
      <line>277</line>
    </frame>
    <frame>
      <ip>0xD78254</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>2175</line>
    </frame>
    <frame>
      <ip>0xD84C00</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1687</line>
    </frame>
    <frame>
      <ip>0xD8DDAC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>51</line>
    </frame>
    <frame>
      <ip>0xC1FEDC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xD35173C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

see http://pulpito.ceph.com/kchai-2017-01-04_05:03:43-rados-wip-kefu-testing---basic-mira/686879/

Actions
Copy link
 #2

Updated by Kefu Chai about 7 years ago 

<error>
  <unique>0x59</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>19,761 (32 direct, 19,729 indirect) bytes in 1 blocks are definitely lost in loss record 90 of 92</text>
    <leakedbytes>19761</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C28203</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>334</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/ext</dir>
      <file>new_allocator.h</file>
      <line>104</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>334</line>
    </frame>

    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>_M_create_node&lt;ceph::buffer::ptr&gt;</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>502</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>_M_insert&lt;ceph::buffer::ptr&gt;</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>1561</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>push_back</fn>
      <dir>/usr/include/c++/4.8.2/bits</dir>
      <file>stl_list.h</file>
      <line>1021</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>push_back</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/include</dir>
      <file>buffer.h</file>
      <line>752</line>
    </frame>
    <frame>
      <ip>0x5ECCA3C</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>ceph::buffer::list::push_back(ceph::buffer::raw*)</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/include</dir>
      <file>buffer.h</file>
      <line>755</line>
    </frame>
    <frame>
      <ip>0x5ECA69E</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>AsyncConnection::process()</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
      <file>AsyncConnection.cc</file>
      <line>572</line>
    </frame>
    <frame>
      <ip>0x5EDA178</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>EventCenter::process_events(int)</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
      <file>Event.cc</file>
      <line>405</line>
    </frame>
    <frame>
      <ip>0x5EDD6E9</ip>
      <obj>/usr/lib64/ceph/libceph-common.so</obj>
      <fn>NetworkStack::add_thread(unsigned int)::{lambda()#1}::operator()() const</fn>
      <dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
      <file>Stack.cc</file>
      <line>46</line>
    </frame>
    <frame>
      <ip>0x1010F22F</ip>
      <obj>/usr/lib64/libstdc++.so.6.0.19</obj>
    </frame>
    <frame>
      <ip>0xF609DC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0x1097273C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

see http://pulpito.ceph.com/kchai-2017-01-22_15:42:56-rados-wip-kefu-testing---basic-mira/738594/

Actions
Copy link
 #3

Updated by Kefu Chai about 7 years ago

both leaks happened at the same place: when the newly allocated buffer (the payload of a message) is appended to the "front" bufferlist. 32 bytes direct leak are contributed by the node of bufferlist::_buffers, and the larger indirect leak is, i think, the internal storage pointing to the allocated buffer for holding the raw buffer.

since it is the list which is holding the node, so the node should be always be freed after the list is destroyed. but in the second report, there is only one leak report. so the list itself is destroyed. weird enough...

Actions
Copy link
 #4

Updated by Josh Durgin almost 7 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF