Project

General

Profile

Bug #54603

Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined".

Added by Sridhar Seshasayee 11 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2022-03-16_20:38:07-rados-wip-yuri3-testing-2022-03-16-1030-distro-default-smithi/6739491

Valgrind reports Leak_IndirectlyLost error on mon.a:

<error>
  <unique>0x2c9d840</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>Leak_IndirectlyLost</kind>
  <xwhat>
    <text>24 bytes in 1 blocks are indirectly lost in loss record 7 of 47</text>
    <leakedbytes>24</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C38B6F</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.18.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>640</line>
    </frame>
    <frame>
      <ip>0x54A8FFB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ceph::buffer::v15_2_0::ptr_node::create_hypercombined(ceph::unique_leakable_ptr&lt;ceph::buffer::v15_2_0::raw&gt;)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/common</dir>
      <file>buffer.cc</file>
      <line>2264</line>
    </frame>
    <frame>
      <ip>0x5431943</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>create</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>377</line>
    </frame>
    <frame>
      <ip>0x5431943</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::read_frame()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1084</line>
    </frame>
    <frame>
      <ip>0x5435D4B</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::run_continuation(Ct&lt;ProtocolV2&gt;&amp;)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>49</line>
    </frame>
    <frame>
      <ip>0x53FC398</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>operator()</fn>
      <dir>/usr/include/c++/8/bits</dir>
      <file>std_function.h</file>
      <line>687</line>
    </frame>
    <frame>
      <ip>0x53FC398</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>AsyncConnection::process()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
      <file>AsyncConnection.cc</file>
      <line>465</line>
    </frame>
    <frame>
      <ip>0x545C206</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>EventCenter::process_events(unsigned int, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
      <file>Event.cc</file>
      <line>422</line>
    </frame>
    <frame>
      <ip>0x5463CCD</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>operator()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
      <file>Stack.cc</file>
      <line>50</line>
    </frame>
    <frame>
      <ip>0x5463CCD</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>std::_Function_handler&lt;void (), NetworkStack::add_thread(Worker*)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;)</fn>
      <dir>/usr/include/c++/8/bits</dir>
      <file>std_function.h</file>
      <line>297</line>
    </frame>
    <frame>
      <ip>0x81EEBA2</ip>
      <obj>/usr/lib64/libstdc++.so.6.0.25</obj>
    </frame>
    <frame>
      <ip>0x784217E</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0x8A94D82</ip>
      <obj>/usr/lib64/libc-2.28.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

Related issues

Related to RADOS - Bug #55809: "Leak_IndirectlyLost" valgrind report on mon.c New

History

#1 Updated by Laura Flores 8 months ago

  • Related to Bug #55809: "Leak_IndirectlyLost" valgrind report on mon.c added

#2 Updated by Sridhar Seshasayee 8 months ago

Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215

Details:

<error>
  <unique>0x2033e52</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>Leak_IndirectlyLost</kind>
  <xwhat>
    <text>24 bytes in 1 blocks are indirectly lost in loss record 6 of 39</text>
    <leakedbytes>24</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C38B6F</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.19.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>640</line>
    </frame>
    <frame>
      <ip>0x5433C06</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ceph::buffer::v15_2_0::ptr_node::cloner::operator()(ceph::buffer::v15_2_0::ptr_node const&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/common</dir>
      <file>buffer.cc</file>
      <line>2228</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>clone_from</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>592</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>operator=</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>963</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>operator=</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
      <file>buffer.h</file>
      <line>960</line>
    </frame>
    <frame>
      <ip>0x4EF992</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServerData::get_caps(ceph::common::CephContext*, EntityName const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>128</line>
    </frame>
    <frame>
      <ip>0x4EFD06</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServer::_get_service_caps(EntityName const&amp;, unsigned int, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>400</line>
    </frame>
    <frame>
      <ip>0x4EFD9D</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>KeyServer::get_service_caps(EntityName const&amp;, unsigned int, AuthCapsInfo&amp;) const</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxKeyServer.cc</file>
      <line>407</line>
    </frame>
    <frame>
      <ip>0x6358CD</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>CephxServiceHandler::handle_request(ceph::buffer::v15_2_0::list::iterator_impl&lt;true&gt;&amp;, unsigned long, ceph::buffer::v15_2_0::list*, AuthCapsInfo*, CryptoKey*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
      <file>CephxServiceHandler.cc</file>
      <line>254</line>
    </frame>
    <frame>
      <ip>0x3D8568</ip>
      <obj>/usr/bin/ceph-mon</obj>
      <fn>Monitor::handle_auth_request(Connection*, AuthConnectionMeta*, bool, unsigned int, ceph::buffer::v15_2_0::list const&amp;, ceph::buffer::v15_2_0::list*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/mon</dir>
      <file>Monitor.cc</file>
      <line>6435</line>
    </frame>
    <frame>
      <ip>0x53EE01E</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_auth_request(ceph::buffer::v15_2_0::list&amp;, bool)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>2155</line>
    </frame>
    <frame>
      <ip>0x53EEFDD</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_auth_request_more(ceph::buffer::v15_2_0::list&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>2216</line>
    </frame>
    <frame>
      <ip>0x53F7A82</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_frame_payload()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1224</line>
    </frame>
    <frame>
      <ip>0x53F7D3F</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_read_frame_dispatch()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1139</line>
    </frame>
    <frame>
      <ip>0x53F7F34</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_read_frame_epilogue_main()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1329</line>
    </frame>
    <frame>
      <ip>0x53F7FE1</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::_handle_read_frame_segment()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1198</line>
    </frame>
    <frame>
      <ip>0x53F9140</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::handle_read_frame_segment(std::unique_ptr&lt;ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer&gt;&amp;&amp;, int)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>1190</line>
    </frame>
    <frame>
      <ip>0x53E18AB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::run_continuation(Ct&lt;ProtocolV2&gt;&amp;)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>47</line>
    </frame>
    <frame>
      <ip>0x53E1A8E</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>ProtocolV2::read_event()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>ProtocolV2.cc</file>
      <line>479</line>
    </frame>
    <frame>
      <ip>0x53A9ACB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>AsyncConnection::process()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>AsyncConnection.cc</file>
      <line>464</line>
    </frame>
    <frame>
      <ip>0x5404755</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>EventCenter::process_events(unsigned int, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>Event.cc</file>
      <line>449</line>
    </frame>
    <frame>
      <ip>0x540A3DB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>operator()</fn>
      <dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
      <file>Stack.cc</file>
      <line>53</line>
    </frame>
    <frame>
      <ip>0x540A3DB</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>std::_Function_handler&lt;void (), NetworkStack::add_thread(unsigned int)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;)</fn>
      <dir>/usr/include/c++/8/bits</dir>
      <file>std_function.h</file>
      <line>297</line>
    </frame>
    <frame>
      <ip>0x10527BA2</ip>
      <obj>/usr/lib64/libstdc++.so.6.0.25</obj>
    </frame>
    <frame>
      <ip>0xFB7B1C9</ip>
      <obj>/usr/lib64/libpthread-2.28.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0x10DCDD82</ip>
      <obj>/usr/lib64/libc-2.28.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
</error>

#3 Updated by Laura Flores 8 months ago

  • Backport set to pacific

/a/yuriw-2022-06-16_19:58:30-rados-wip-yuri7-testing-2022-06-16-1051-pacific-distro-default-smithi/6883120

#4 Updated by Laura Flores 6 months ago

/a/yuriw-2022-08-11_16:46:00-rados-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/6967922/remote/smithi059/log/valgrind/mon.a.log.gz

#5 Updated by Laura Flores 6 months ago

Sridhar Seshasayee wrote:

Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215

Details:

[...]

This looks like a different issue with "KeyServerData::get_caps" (the other occurrences on this Tracker concern "buffer::v15_2_0::ptr_node::create_hypercombined".

I'm seeing another occurrence of "KeyServerData::get_caps" here, so let's create another Tracker for it:
/a/yuriw-2022-08-19_20:57:42-rados-wip-yuri6-testing-2022-08-19-0940-pacific-distro-default-smithi/6981517/remote/smithi130/log/valgrind/mon.a.log.gz

#6 Updated by Laura Flores 6 months ago

  • Subject changed from Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon. to Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined".

Also available in: Atom PDF