Project

General

Profile

Bug #52136

Valgrind reports memory "Leak_DefinitelyLost" errors.

Added by Sridhar Seshasayee over 2 years ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
low-hanging-fruit backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Valgrind reported the memory leak error in the following jobs:

/a/yuriw-2021-08-05_21:11:40-rados-wip-yuri-testing-2021-08-05-0936-distro-basic-smithi/6315879
/a/yuriw-2021-08-05_21:11:40-rados-wip-yuri-testing-2021-08-05-0936-distro-basic-smithi/6316099
/a/yuriw-2021-08-10_20:56:51-rados-wip-yuri-testing-2021-08-10-1139-distro-basic-smithi/6332241

Following is the error reported from 6315879 on osd.0:

<error>
  <unique>0x51c2a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>8 bytes in 1 blocks are definitely lost in loss record 140 of 3,017</text>
    <leakedbytes>8</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0x4C31C93</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>431</line>
    </frame>
    <frame>
      <ip>0x1B9EF17</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionStorageInfo::VersionStorageInfo(rocksdb::InternalKeyComparator const*, rocksdb::Comparator const*, int, rocksdb::CompactionStyle, rocksdb::VersionStorageInfo*, bool)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_set.cc</file>
      <line>1724</line>
    </frame>
    <frame>
      <ip>0x1B9F685</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::Version::Version(rocksdb::ColumnFamilyData*, rocksdb::VersionSet*, rocksdb::FileOptions const&amp;, rocksdb::MutableCFOptions, std::shared_ptr&lt;rocksdb::IOTracer&gt; const&amp;, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_set.cc</file>
      <line>1791</line>
    </frame>
    <frame>
      <ip>0x1BB55AD</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionSet::CreateColumnFamily(rocksdb::ColumnFamilyOptions const&amp;, rocksdb::VersionEdit const*)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_set.cc</file>
      <line>5764</line>
    </frame>
    <frame>
      <ip>0x1E095F5</ip>
     <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionEditHandler::CreateCfAndInit(rocksdb::ColumnFamilyOptions const&amp;, rocksdb::VersionEdit const&amp;)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_edit_handler.cc</file>
      <line>409</line>
    </frame>
    <frame>
      <ip>0x1E07BDC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionEditHandler::Initialize()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_edit_handler.cc</file>
      <line>143</line>
    </frame>
    <frame>
      <ip>0x1E06F53</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionEditHandlerBase::Iterate(rocksdb::log::Reader&amp;, rocksdb::Status*)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_edit_handler.cc</file>
      <line>26</line>
    </frame>
    <frame>
      <ip>0x1BAF7AC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::VersionSet::Recover(std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, bool, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;*)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
      <file>version_set.cc</file>
      <line>4752</line>
    </frame>
    <frame>
      <ip>0x1ACCFE7</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::DBImpl::Recover(std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, bool, bool, bool, unsigned long*)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
      <file>db_impl_open.cc</file>
      <line>473</line>
    </frame>
    <frame>
      <ip>0x1AD3996</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**, bool, bool)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
      <file>db_impl_open.cc</file>
     <line>1554</line>
    </frame>
    <frame>
      <ip>0x1AD2D03</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>rocksdb::DB::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
      <file>db_impl_open.cc</file>
      <line>1448</line>
    </frame>
    <frame>
      <ip>0x19D7E4A</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>RocksDBStore::do_open(std::ostream&amp;, bool, bool, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/kv</dir>
      <file>RocksDBStore.cc</file>
      <line>1152</line>
    </frame>
    <frame>
      <ip>0x19EFA07</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>RocksDBStore::open(std::ostream&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/kv</dir>
      <file>RocksDBStore.h</file>
      <line>258</line>
    </frame>
    <frame>
      <ip>0x138573D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::_open_db(bool, bool, bool)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>6253</line>
    </frame>
    <frame>
      <ip>0x13FBDFD</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::_open_db_and_around(bool, bool)</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>6012</line>
    </frame>
    <frame>
      <ip>0x13FE495</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::_mount()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
      <file>BlueStore.cc</file>
      <line>7171</line>
    </frame>
    <frame>
      <ip>0x14334CE</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>BlueStore::mount()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
      <file>BlueStore.h</file>
      <line>2623</line>
    </frame>
    <frame>
      <ip>0xE3AB1D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::init()</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/osd</dir>
      <file>OSD.cc</file>
      <line>3554</line>
    </frame>
    <frame>
      <ip>0xDB1BC2</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>main</fn>
      <dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src</dir>
      <file>ceph_osd.cc</file>
      <line>711</line>
    </frame>
  </stack>
</error>


Related issues

Related to RADOS - Bug #57751: LibRadosAio.SimpleWritePP hang and pkill Resolved
Related to RADOS - Bug #57618: rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify) Resolved
Copied to RADOS - Backport #58314: pacific: Valgrind reports memory "Leak_DefinitelyLost" errors. Resolved
Copied to RADOS - Backport #58315: quincy: Valgrind reports memory "Leak_DefinitelyLost" errors. Resolved

History

#1 Updated by Adam Kupczyk over 2 years ago

This leak is from internals of RocksDB.
We have no access to FileMetaData objects, we cannot be responsible for this leak.
We should either forward this to RocksDB developers, or ignore (my preferred option).

#2 Updated by Neha Ojha over 2 years ago

Adam Kupczyk wrote:

This leak is from internals of RocksDB.
We have no access to FileMetaData objects, we cannot be responsible for this leak.
We should either forward this to RocksDB developers, or ignore (my preferred option).

we could add it to the valgrind suppression file.

#3 Updated by Laura Flores about 2 years ago

/a/yuriw-2021-12-09_00:18:57-rados-wip-yuri-testing-2021-12-08-1336-distro-default-smithi/6553724/ ----> osd.1.log.gz

#4 Updated by Laura Flores about 2 years ago

/a/sseshasa-2022-02-24_11:27:07-rados-wip-45118-45121-quincy-testing-distro-default-smithi/6704275/remote/smithi174/log/valgrind/osd.6.log.gz

#5 Updated by Neha Ojha almost 2 years ago

  • Tags set to low-hanging-fruit

Let's add it to qa/valgrind.supp to suppress this error, based on Adam's comment https://tracker.ceph.com/issues/52136#note-1.

#6 Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-03-26_19:43:35-rados-wip-yuri7-testing-2022-03-24-1341-pacific-distro-default-smithi/6762662

#7 Updated by Laura Flores over 1 year ago

  • Tags set to low-hanging-fruit

#8 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962388/

#9 Updated by Radoslaw Zarzynski over 1 year ago

How about adding to suppression file regarding the comment #1?

#10 Updated by Laura Flores over 1 year ago

  • Tags deleted (low-hanging-fruit)

#11 Updated by Nitzan Mordechai over 1 year ago

  • Related to Bug #57751: LibRadosAio.SimpleWritePP hang and pkill added

#12 Updated by Nitzan Mordechai over 1 year ago

  • Assignee set to Nitzan Mordechai

#13 Updated by Nitzan Mordechai over 1 year ago

  • Status changed from New to In Progress

#14 Updated by Neha Ojha over 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 48641

#15 Updated by Nitzan Mordechai over 1 year ago

  • Related to Bug #57618: rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify) added

#17 Updated by Radoslaw Zarzynski about 1 year ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific,quincy

#18 Updated by Backport Bot about 1 year ago

  • Copied to Backport #58314: pacific: Valgrind reports memory "Leak_DefinitelyLost" errors. added

#19 Updated by Backport Bot about 1 year ago

  • Copied to Backport #58315: quincy: Valgrind reports memory "Leak_DefinitelyLost" errors. added

#20 Updated by Backport Bot about 1 year ago

  • Tags changed from low-hanging-fruit to low-hanging-fruit backport_processed

#21 Updated by Nitzan Mordechai 7 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF