Actions
Bug #54603
openValgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined".
Status:
New
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/yuriw-2022-03-16_20:38:07-rados-wip-yuri3-testing-2022-03-16-1030-distro-default-smithi/6739491
Valgrind reports Leak_IndirectlyLost error on mon.a:
<error>
<unique>0x2c9d840</unique>
<tid>1</tid>
<threadname>ceph-mon</threadname>
<kind>Leak_IndirectlyLost</kind>
<xwhat>
<text>24 bytes in 1 blocks are indirectly lost in loss record 7 of 47</text>
<leakedbytes>24</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4C38B6F</ip>
<obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new[](unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.18.1/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>640</line>
</frame>
<frame>
<ip>0x54A8FFB</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ceph::buffer::v15_2_0::ptr_node::create_hypercombined(ceph::unique_leakable_ptr<ceph::buffer::v15_2_0::raw>)</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/common</dir>
<file>buffer.cc</file>
<line>2264</line>
</frame>
<frame>
<ip>0x5431943</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>create</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/include</dir>
<file>buffer.h</file>
<line>377</line>
</frame>
<frame>
<ip>0x5431943</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::read_frame()</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1084</line>
</frame>
<frame>
<ip>0x5435D4B</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::run_continuation(Ct<ProtocolV2>&)</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>49</line>
</frame>
<frame>
<ip>0x53FC398</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>operator()</fn>
<dir>/usr/include/c++/8/bits</dir>
<file>std_function.h</file>
<line>687</line>
</frame>
<frame>
<ip>0x53FC398</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>AsyncConnection::process()</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
<file>AsyncConnection.cc</file>
<line>465</line>
</frame>
<frame>
<ip>0x545C206</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
<file>Event.cc</file>
<line>422</line>
</frame>
<frame>
<ip>0x5463CCD</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>operator()</fn>
<dir>/usr/src/debug/ceph-17.0.0-11135.gbf232af4.el8.x86_64/src/msg/async</dir>
<file>Stack.cc</file>
<line>50</line>
</frame>
<frame>
<ip>0x5463CCD</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>std::_Function_handler<void (), NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data const&)</fn>
<dir>/usr/include/c++/8/bits</dir>
<file>std_function.h</file>
<line>297</line>
</frame>
<frame>
<ip>0x81EEBA2</ip>
<obj>/usr/lib64/libstdc++.so.6.0.25</obj>
</frame>
<frame>
<ip>0x784217E</ip>
<obj>/usr/lib64/libpthread-2.28.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0x8A94D82</ip>
<obj>/usr/lib64/libc-2.28.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
Updated by Laura Flores almost 2 years ago
- Related to Bug #55809: "Leak_IndirectlyLost" valgrind report on mon.c added
Updated by Sridhar Seshasayee almost 2 years ago
Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215
Details:
<error>
<unique>0x2033e52</unique>
<tid>1</tid>
<threadname>ceph-mon</threadname>
<kind>Leak_IndirectlyLost</kind>
<xwhat>
<text>24 bytes in 1 blocks are indirectly lost in loss record 6 of 39</text>
<leakedbytes>24</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4C38B6F</ip>
<obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new[](unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.19.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>640</line>
</frame>
<frame>
<ip>0x5433C06</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ceph::buffer::v15_2_0::ptr_node::cloner::operator()(ceph::buffer::v15_2_0::ptr_node const&)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/common</dir>
<file>buffer.cc</file>
<line>2228</line>
</frame>
<frame>
<ip>0x4EF992</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>clone_from</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
<file>buffer.h</file>
<line>592</line>
</frame>
<frame>
<ip>0x4EF992</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>operator=</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
<file>buffer.h</file>
<line>963</line>
</frame>
<frame>
<ip>0x4EF992</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>operator=</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/include</dir>
<file>buffer.h</file>
<line>960</line>
</frame>
<frame>
<ip>0x4EF992</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>KeyServerData::get_caps(ceph::common::CephContext*, EntityName const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, AuthCapsInfo&) const</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
<file>CephxKeyServer.cc</file>
<line>128</line>
</frame>
<frame>
<ip>0x4EFD06</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>KeyServer::_get_service_caps(EntityName const&, unsigned int, AuthCapsInfo&) const</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
<file>CephxKeyServer.cc</file>
<line>400</line>
</frame>
<frame>
<ip>0x4EFD9D</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>KeyServer::get_service_caps(EntityName const&, unsigned int, AuthCapsInfo&) const</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
<file>CephxKeyServer.cc</file>
<line>407</line>
</frame>
<frame>
<ip>0x6358CD</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>CephxServiceHandler::handle_request(ceph::buffer::v15_2_0::list::iterator_impl<true>&, unsigned long, ceph::buffer::v15_2_0::list*, AuthCapsInfo*, CryptoKey*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/auth/cephx</dir>
<file>CephxServiceHandler.cc</file>
<line>254</line>
</frame>
<frame>
<ip>0x3D8568</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>Monitor::handle_auth_request(Connection*, AuthConnectionMeta*, bool, unsigned int, ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list*)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/mon</dir>
<file>Monitor.cc</file>
<line>6435</line>
</frame>
<frame>
<ip>0x53EE01E</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::_handle_auth_request(ceph::buffer::v15_2_0::list&, bool)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>2155</line>
</frame>
<frame>
<ip>0x53EEFDD</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::handle_auth_request_more(ceph::buffer::v15_2_0::list&)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>2216</line>
</frame>
<frame>
<ip>0x53F7A82</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::handle_frame_payload()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1224</line>
</frame>
<frame>
<ip>0x53F7D3F</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::handle_read_frame_dispatch()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1139</line>
</frame>
<frame>
<ip>0x53F7F34</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::_handle_read_frame_epilogue_main()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1329</line>
</frame>
<frame>
<ip>0x53F7FE1</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::_handle_read_frame_segment()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1198</line>
</frame>
<frame>
<ip>0x53F9140</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::handle_read_frame_segment(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>1190</line>
</frame>
<frame>
<ip>0x53E18AB</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::run_continuation(Ct<ProtocolV2>&)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>47</line>
</frame>
<frame>
<ip>0x53E1A8E</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>ProtocolV2::read_event()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>ProtocolV2.cc</file>
<line>479</line>
</frame>
<frame>
<ip>0x53A9ACB</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>AsyncConnection::process()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>AsyncConnection.cc</file>
<line>464</line>
</frame>
<frame>
<ip>0x5404755</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>Event.cc</file>
<line>449</line>
</frame>
<frame>
<ip>0x540A3DB</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>operator()</fn>
<dir>/usr/src/debug/ceph-16.2.9-239.g224fc22e.el8.x86_64/src/msg/async</dir>
<file>Stack.cc</file>
<line>53</line>
</frame>
<frame>
<ip>0x540A3DB</ip>
<obj>/usr/lib64/ceph/libceph-common.so.2</obj>
<fn>std::_Function_handler<void (), NetworkStack::add_thread(unsigned int)::{lambda()#1}>::_M_invoke(std::_Any_data const&)</fn>
<dir>/usr/include/c++/8/bits</dir>
<file>std_function.h</file>
<line>297</line>
</frame>
<frame>
<ip>0x10527BA2</ip>
<obj>/usr/lib64/libstdc++.so.6.0.25</obj>
</frame>
<frame>
<ip>0xFB7B1C9</ip>
<obj>/usr/lib64/libpthread-2.28.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0x10DCDD82</ip>
<obj>/usr/lib64/libc-2.28.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
Updated by Laura Flores almost 2 years ago
- Backport set to pacific
/a/yuriw-2022-06-16_19:58:30-rados-wip-yuri7-testing-2022-06-16-1051-pacific-distro-default-smithi/6883120
Updated by Laura Flores over 1 year ago
/a/yuriw-2022-08-11_16:46:00-rados-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/6967922/remote/smithi059/log/valgrind/mon.a.log.gz
Updated by Laura Flores over 1 year ago
Sridhar Seshasayee wrote:
Seen on mon.b in pacific run:
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881215Details:
[...]
This looks like a different issue with "KeyServerData::get_caps" (the other occurrences on this Tracker concern "buffer::v15_2_0::ptr_node::create_hypercombined".
I'm seeing another occurrence of "KeyServerData::get_caps" here, so let's create another Tracker for it:
/a/yuriw-2022-08-19_20:57:42-rados-wip-yuri6-testing-2022-08-19-0940-pacific-distro-default-smithi/6981517/remote/smithi130/log/valgrind/mon.a.log.gz
Updated by Laura Flores over 1 year ago
- Subject changed from Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon. to Valgrind reports memory "Leak_IndirectlyLost" errors on ceph-mon in "buffer::ptr_node::create_hypercombined".
Updated by Laura Flores about 1 year ago
- Translation missing: en.field_tag_list set to test-failure
/a/lflores-2023-02-06_22:55:02-rados-wip-yuri10-testing-2023-02-06-0846-distro-default-smithi/7151905
Updated by Laura Flores 10 months ago
/a/yuriw-2023-06-22_20:09:40-rados-wip-yuri6-testing-2023-06-22-1005-quincy-distro-default-smithi/7312628
Updated by Laura Flores 10 months ago
- Backport changed from pacific to pacific,quincy
Actions