Actions
Bug #17417
closed/var/log/ceph/valgrind/osd.1.log: <kind>Leak_DefinitelyLost</kind>
Added by David Zafman over 7 years ago. Updated almost 7 years ago.
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Updated by Kefu Chai over 7 years ago
the log is gone, this is a new occurrence.
<error>
<unique>0x55</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>1,041 (32 direct, 1,009 indirect) bytes in 1 blocks are definitely lost in loss record 86 of 94</text>
<leakedbytes>1041</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0xA0EB203</ip>
<obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new(unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>334</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/ext</dir>
<file>new_allocator.h</file>
<line>104</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>334</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>502</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>1561</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>1021</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>push_back</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/include</dir>
<file>buffer.h</file>
<line>752</line>
</frame>
<frame>
<ip>0xD77E32</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>2077</line>
</frame>
<frame>
<ip>0xD84C00</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::reader()</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>1687</line>
</frame>
<frame>
<ip>0xD8DDAC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::Reader::entry()</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.h</file>
<line>51</line>
</frame>
<frame>
<ip>0xC1FEDC4</ip>
<obj>/usr/lib64/libpthread-2.17.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0xD35173C</ip>
<obj>/usr/lib64/libc-2.17.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
and
<error>
<unique>0x56</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>1,064 (56 direct, 1,008 indirect) bytes in 1 blocks are definitely lost in loss record 87 of 94</text>
<leakedbytes>1064</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0xA0EB203</ip>
<obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new(unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>334</line>
</frame>
<frame>
<ip>0xD4118C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/ext</dir>
<file>new_allocator.h</file>
<line>104</line>
</frame>
<frame>
<ip>0xD4118C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>_M_get_node</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_tree.h</file>
<line>370</line>
</frame>
<frame>
<ip>0xD4118C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>_M_create_node<const std::piecewise_construct_t&, std::tuple<const std::pair<long unsigned int, long unsigned int>&>, std::tuple<> ></fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_tree.h</file>
<line>403</line>
</frame>
<frame>
<ip>0xD4118C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>std::_Rb_tree_iterator<std::pair<std::pair<unsigned long, unsigned long> const, std::pair<unsigned int, unsigned int> > > std::_Rb_tree<std::pair<unsigned long, unsigned long>, std::pair<std::pair<unsigned long, unsigned long> const, std::pair<unsigned int, unsigned int> >, std::_Select1st<std::pair<std::pair<unsigned long, unsigned long> const, std::pair<unsigned int, unsigned int> > >, std::less<std::pair<unsigned long, unsigned long> >, std::allocator<std::pair<std::pair<unsigned long, unsigned long> const, std::pair<unsigned int, unsigned int> > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::pair<unsigned long, unsigned long> const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::pair<unsigned long, unsigned long> const, std::pair<unsigned int, unsigned int> > >, std::piecewise_construct_t const&, std::tuple<std::pair<unsigned long, unsigned long> const&>&&, std::tuple<>&&)</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_tree.h</file>
<line>1669</line>
</frame>
<frame>
<ip>0xD3D1C0</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>set_crc</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/common</dir>
<file>buffer.cc</file>
<line>229</line>
</frame>
<frame>
<ip>0xD3D1C0</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>ceph::buffer::list::crc32c(unsigned int) const</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/common</dir>
<file>buffer.cc</file>
<line>2411</line>
</frame>
<frame>
<ip>0xBF0011</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&)</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg</dir>
<file>Message.cc</file>
<line>277</line>
</frame>
<frame>
<ip>0xD78254</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>2175</line>
</frame>
<frame>
<ip>0xD84C00</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::reader()</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>1687</line>
</frame>
<frame>
<ip>0xD8DDAC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::Reader::entry()</fn>
<dir>/usr/src/debug/ceph-11.1.0-6391-ge136ad6/src/msg/simple</dir>
<file>Pipe.h</file>
<line>51</line>
</frame>
<frame>
<ip>0xC1FEDC4</ip>
<obj>/usr/lib64/libpthread-2.17.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0xD35173C</ip>
<obj>/usr/lib64/libc-2.17.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
see http://pulpito.ceph.com/kchai-2017-01-04_05:03:43-rados-wip-kefu-testing---basic-mira/686879/
Updated by Kefu Chai over 7 years ago
<error>
<unique>0x59</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>19,761 (32 direct, 19,729 indirect) bytes in 1 blocks are definitely lost in loss record 90 of 92</text>
<leakedbytes>19761</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4C28203</ip>
<obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new(unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>334</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/ext</dir>
<file>new_allocator.h</file>
<line>104</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>UnknownInlinedFun</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>334</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>_M_create_node<ceph::buffer::ptr></fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>502</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>_M_insert<ceph::buffer::ptr></fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>1561</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>push_back</fn>
<dir>/usr/include/c++/4.8.2/bits</dir>
<file>stl_list.h</file>
<line>1021</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>push_back</fn>
<dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/include</dir>
<file>buffer.h</file>
<line>752</line>
</frame>
<frame>
<ip>0x5ECCA3C</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>ceph::buffer::list::push_back(ceph::buffer::raw*)</fn>
<dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/include</dir>
<file>buffer.h</file>
<line>755</line>
</frame>
<frame>
<ip>0x5ECA69E</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>AsyncConnection::process()</fn>
<dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
<file>AsyncConnection.cc</file>
<line>572</line>
</frame>
<frame>
<ip>0x5EDA178</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>EventCenter::process_events(int)</fn>
<dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
<file>Event.cc</file>
<line>405</line>
</frame>
<frame>
<ip>0x5EDD6E9</ip>
<obj>/usr/lib64/ceph/libceph-common.so</obj>
<fn>NetworkStack::add_thread(unsigned int)::{lambda()#1}::operator()() const</fn>
<dir>/usr/src/debug/ceph-11.1.0-6911-g9eb7592/src/msg/async</dir>
<file>Stack.cc</file>
<line>46</line>
</frame>
<frame>
<ip>0x1010F22F</ip>
<obj>/usr/lib64/libstdc++.so.6.0.19</obj>
</frame>
<frame>
<ip>0xF609DC4</ip>
<obj>/usr/lib64/libpthread-2.17.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0x1097273C</ip>
<obj>/usr/lib64/libc-2.17.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
see http://pulpito.ceph.com/kchai-2017-01-22_15:42:56-rados-wip-kefu-testing---basic-mira/738594/
Updated by Kefu Chai over 7 years ago
both leaks happened at the same place: when the newly allocated buffer (the payload of a message) is appended to the "front" bufferlist. 32 bytes direct leak are contributed by the node of bufferlist::_buffers, and the larger indirect leak is, i think, the internal storage pointing to the allocated buffer for holding the raw buffer.
since it is the list which is holding the node, so the node should be always be freed after the list is destroyed. but in the second report, there is only one leak report. so the list itself is destroyed. weird enough...
Updated by Josh Durgin almost 7 years ago
- Status changed from New to Can't reproduce
Actions