Actions
Bug #58925
closedrocksdb "Leak_StillReachable" memory leak in mons
% Done:
0%
Source:
Tags:
backport_processed
Backport:
reef
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
<error>
<unique>0x25b8e7</unique>
<tid>1</tid>
<threadname>ceph-mon</threadname>
<kind>Leak_StillReachable</kind>
<xwhat>
<text>20 bytes in 1 blocks are still reachable in loss record 17 of 173</text>
<leakedbytes>20</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4C38B6F</ip>
<obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new[](unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.19.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>640</line>
</frame>
<frame>
<ip>0xA28BCD</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0]</fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits</dir>
<file>basic_string.tcc</file>
<line>219</line>
</frame>
<frame>
<ip>0xA2B68A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>UnknownInlinedFun</fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits</dir>
<file>basic_string.h</file>
<line>255</line>
</frame>
<frame>
<ip>0xA2B68A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>UnknownInlinedFun</fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits</dir>
<file>basic_string.h</file>
<line>274</line>
</frame>
<frame>
<ip>0xA2B68A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>UnknownInlinedFun</fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits</dir>
<file>basic_string.h</file>
<line>459</line>
</frame>
<frame>
<ip>0xA2B68A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>PatternEntry</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/include/rocksdb/utilities</dir>
<file>object_registry.h</file>
<line>73</line>
</frame>
<frame>
<ip>0xA2B68A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>std::function<rocksdb::SliceTransform const* (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<rocksdb::SliceTransform const, std::default_delete<rocksdb::SliceTransform const> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)> const& rocksdb::ObjectLibrary::AddFactory<rocksdb::SliceTransform const>(rocksdb::ObjectLibrary::PatternEntry const&, std::function<rocksdb::SliceTransform const* (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<rocksdb::SliceTransform const, std::default_delete<rocksdb::SliceTransform const> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)> const&) [clone .isra.0]</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/include/rocksdb/utilities</dir>
<file>object_registry.h</file>
<line>255</line>
</frame>
<frame>
<ip>0xA2C195</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::RegisterBuiltinSliceTransform(rocksdb::ObjectLibrary&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .constprop.0] [clone .isra.0]</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/util</dir>
<file>slice.cc</file>
<line>182</line>
</frame>
<frame>
<ip>0xA2C95B</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>std::once_flag::_Prepare_execution::_Prepare_execution<std::call_once<rocksdb::SliceTransform::CreateFromString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rocksdb::SliceTransform const>*)::{lambda()#1}>(std::once_flag&, rocksdb::SliceTransform::CreateFromString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rocksdb::SliceTransform const>*)::{lambda()#1}&&)::{lambda()#1}>(rocksdb::SliceTransform::CreateFromString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rocksdb::SliceTransform const>*)::{lambda()#1}&)::{lambda()#1}::_FUN()</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/util</dir>
<file>slice.cc</file>
<line>223</line>
</frame>
<frame>
<ip>0x7DDDE66</ip>
<obj>/usr/lib64/libpthread-2.28.so</obj>
<fn>__pthread_once_slow</fn>
</frame>
<frame>
<ip>0xA2B250</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>UnknownInlinedFun</fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11/x86_64-redhat-linux/bits</dir>
<file>gthr-default.h</file>
<line>700</line>
</frame>
<frame>
<ip>0xA2B250</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>call_once<rocksdb::SliceTransform::CreateFromString(const rocksdb::ConfigOptions&, const string&, std::shared_ptr<const rocksdb::SliceTransform>*)::<lambda()> ></fn>
<dir>/opt/rh/gcc-toolset-11/root/usr/include/c++/11</dir>
<file>mutex</file>
<line>783</line>
</frame>
<frame>
<ip>0xA2B250</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::SliceTransform::CreateFromString(rocksdb::ConfigOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rocksdb::SliceTransform const>*)</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/util</dir>
<file>slice.cc</file>
<line>222</line>
</frame>
<frame>
<ip>0x9A58CC</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::CacheReservationManager>, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::array<unsigned long, 2ul>)</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/table/block_based</dir>
<file>block_based_table_reader.cc</file>
<line>754</line>
</frame>
<frame>
<ip>0x994102</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader> >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, bool) const</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/table/block_based</dir>
<file>block_based_table_factory.cc</file>
<line>633</line>
</frame>
<frame>
<ip>0x8A067D</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl*, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader> >*, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature)</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/db</dir>
<file>table_cache.cc</file>
<line>166</line>
</frame>
<frame>
<ip>0x8A0DE1</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Cache::Handle**, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature)</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/db</dir>
<file>table_cache.cc</file>
<line>220</line>
</frame>
<frame>
<ip>0xAE6F4A</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*, int, bool, bool, std::shared_ptr<rocksdb::SliceTransform const> const&, unsigned long)::{lambda()#1}::operator()() const</fn>
<dir>/usr/src/debug/ceph-18.0.0-2694.g33b4b31b.el8.x86_64/src/rocksdb/db</dir>
<file>version_builder.cc</file>
<line>1275</line>
</frame>
<frame>
<ip>0x8783BA2</ip>
<obj>/usr/lib64/libstdc++.so.6.0.25</obj>
</frame>
<frame>
<ip>0x7DD61C9</ip>
<obj>/usr/lib64/libpthread-2.28.so</obj>
<fn>start_thread</fn>
</frame>
<frame>
<ip>0x9029E72</ip>
<obj>/usr/lib64/libc-2.28.so</obj>
<fn>clone</fn>
</frame>
</stack>
</error>
Updated by Laura Flores about 1 year ago
- Backport set to reef
Laura Flores wrote:
[...]
This example was from /a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7193033, a centos stream 8 test.
Updated by Laura Flores about 1 year ago
Caused by https://github.com/ceph/ceph/pull/49006.
Updated by Laura Flores about 1 year ago
Steps to reproduce:
$ cd ceph
$ ninja vstart -j$(nproc)
$ ../src/vstart.sh --debug --new -x --localhost --bluestore
$ mkdir /home/lflores/tmp/
$ mkdir /home/lflores/tmp/valgrind
$ touch /home/lflores/tmp/valgrind/mon.a.log
$ $ ps waxu | grep ceph-mon
lflores 470790 1.7 0.0 460128 136200 ? Ssl 20:43 0:00 /home/lflores/ceph/build/bin/ceph-mon -i a -c /home/lflores/ceph/build/ceph.conf
lflores 470831 1.2 0.0 457064 134012 ? Ssl 20:43 0:00 /home/lflores/ceph/build/bin/ceph-mon -i b -c /home/lflores/ceph/build/ceph.conf
lflores 470872 1.4 0.0 458088 135164 ? Ssl 20:43 0:00 /home/lflores/ceph/build/bin/ceph-mon -i c -c /home/lflores/ceph/build/ceph.conf
$ kill 470790
$ OPENSSL_ia32cap='~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/lflores/ceph/qa/valgrind.supp --xml=yes --xml-file=/home/lflores/tmp/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=no --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes /home/lflores/ceph/build/bin/ceph-mon -i a -c /home/lflores/ceph/build/ceph.conf -d
Updated by Laura Flores about 1 year ago
- Status changed from New to Fix Under Review
- Assignee set to Laura Flores
- Pull request ID set to 50410
Updated by Laura Flores about 1 year ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot about 1 year ago
- Copied to Backport #58979: reef: rocksdb "Leak_StillReachable" memory leak in mons added
Updated by Yuri Weinstein about 1 year ago
Updated by Laura Flores 11 months ago
- Status changed from Pending Backport to Resolved
Actions