Bug #54603: Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined". - Ceph - Ceph

Actions

Copy link

#1

Updated by Laura Flores almost 2 years ago

Related to Bug #55809: "Leak_IndirectlyLost" valgrind report on mon.c added

Actions

Copy link

#2

Updated by Sridhar Seshasayee almost 2 years ago

Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215

Details:

<error>
  <unique>0x2033e52</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>Leak_IndirectlyLost</kind>
  <xwhat>
    <text>24 bytes in 1 blocks are indirectly lost in loss record 6 of 39</text>
    <leakedbytes>24</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C38B6F</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.19.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>640</line>
    </frame>
    <frame>
      <ip>0x5433C06</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ceph::buffer::v15_2_0::ptr_node::cloner::operator()(ceph::buffer::v15_2_0::ptr_node const&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/common</dir>
      <file>buffer.cc</file>
      <line>2228</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>clone_from</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>592</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>operator=</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>963</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>operator=</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>960</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServerData::get_caps(ceph::common::CephContext*, EntityName const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>128</line>
    </frame>
    <frame>
      <ip>0x4EFD06</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServer::_get_service_caps(EntityName const&amp;, unsigned int, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>400</line>
    </frame>
    <frame>
      <ip>0x4EFD9D</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServer::get_service_caps(EntityName const&amp;, unsigned int, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>407</line>
    </frame>
    <frame>
      <ip>0x6358CD</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>CephxServiceHandler::handle_request(ceph::buffer::v15_2_0::list::iterator_impl&lt;true&gt;&amp;, unsigned long, ceph::buffer::v15_2_0::list*, AuthCapsInfo*, CryptoKey*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxServiceHandler.cc</file>
      <line>254</line>
    </frame>
    <frame>
      <ip>0x3D8568</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>Monitor::handle_auth_request(Connection*, AuthConnectionMeta*, bool, unsigned int, ceph::buffer::v15_2_0::list const&amp;, ceph::buffer::v15_2_0::list*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/mon</dir>
      <file>Monitor.cc</file>
      <line>6435</line>
    </frame>
    <frame>
      <ip>0x53EE01E</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_auth_request(ceph::buffer::v15_2_0::list&amp;, bool)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>2155</line>
    </frame>
    <frame>
      <ip>0x53EEFDD</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_auth_request_more(ceph::buffer::v15_2_0::list&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>2216</line>
    </frame>
    <frame>
      <ip>0x53F7A82</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_frame_payload()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1224</line>
    </frame>
    <frame>
      <ip>0x53F7D3F</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_read_frame_dispatch()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1139</line>
    </frame>
    <frame>
      <ip>0x53F7F34</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_read_frame_epilogue_main()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1329</line>
    </frame>
    <frame>
      <ip>0x53F7FE1</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_read_frame_segment()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1198</line>
    </frame>
    <frame>
      <ip>0x53F9140</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_read_frame_segment(std::unique_ptr&lt;ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer&gt;&amp;&amp;, int)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1190</line>
    </frame>
    <frame>
      <ip>0x53E18AB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::run_continuation(Ct&lt;ProtocolV2&gt;&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>47</line>
    </frame>
    <frame>
      <ip>0x53E1A8E</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::read_event()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>479</line>
    </frame>
    <frame>
      <ip>0x53A9ACB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>AsyncConnection::process()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>AsyncConnection.cc</file>
      <line>464</line>
    </frame>
    <frame>
      <ip>0x5404755</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>EventCenter::process_events(unsigned int, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>Event.cc</file>
      <line>449</line>
    </frame>
    <frame>
      <ip>0x540A3DB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>operator()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>Stack.cc</file>
      <line>53</line>
    </frame>
    <frame>
      <ip>0x540A3DB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>std::_Function_handler&lt;void (), NetworkStack::add_thread(unsigned int)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;)</fn>
      <dir>/usr/include/c++/8/bits</dir>
      <file>std_function.h</file>
      <line>297</line>
    </frame>
    <frame>
      <ip>0x10527BA2</ip>
      <obj>/usr/lib64/libstdc++.so.6.0.25</obj>
    </frame>
    <frame>
      <ip>0xFB7B1C9</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0x10DCDD82</ip>
      <obj>/usr/lib64/libc-2.28.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

Actions

Copy link

#3

Updated by Laura Flores almost 2 years ago

Backport set to pacific

/a/yuriw-2022-06-16_19:58:30-rados-wip-yuri7-testing-2022-06-16-1051-pacific-distro-default-smithi/6883120

Actions

Copy link

#4

Updated by Laura Flores over 1 year ago

/a/yuriw-2022-08-11_16:46:00-rados-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/6967922/remote/smithi059/log/valgrind/mon.a.log.gz

Actions

Copy link

#5

Updated by Laura Flores over 1 year ago

Sridhar Seshasayee wrote:

Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215

Details:

[...]

This looks like a different issue with "KeyServerData::get_caps" (the other occurrences on this Tracker concern "buffer::v15_2_0::ptr_node::create_hypercombined".

I'm seeing another occurrence of "KeyServerData::get_caps" here, so let's create another Tracker for it:
/a/yuriw-2022-08-19_20:57:42-rados-wip-yuri6-testing-2022-08-19-0940-pacific-distro-default-smithi/6981517/remote/smithi130/log/valgrind/mon.a.log.gz

Actions

Copy link

#6

Updated by Laura Flores over 1 year ago

Subject changed from Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon. to Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined".

Actions

Copy link

#7