Project

General

Profile

Bug #20491

objecter leaked OSDMap in handle_osd_map

Added by Kefu Chai almost 3 years ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature:

Description

<error>
  <unique>0x4a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>7,638 (816 direct, 6,822 indirect) bytes in 1 blocks are definitely lost in loss record 75 of 78</text>
    <leakedbytes>7638</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA2A58A8</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>423</line>
    </frame>
    <frame>
      <ip>0x5FC613</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Objecter</fn>
      <dir>/usr/src/debug/ceph-12.0.3-2488-g7d2854b/src/osdc</dir>
      <file>Objecter.h</file>
      <line>2012</line>
    </frame>
    <frame>
      <ip>0x5FC613</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSDService::OSDService(OSD*)</fn>
      <dir>/usr/src/debug/ceph-12.0.3-2488-g7d2854b/src/osd</dir>
      <file>OSD.cc</file>
      <line>236</line>
    </frame>
    <frame>
      <ip>0x607081</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::OSD(CephContext*, ObjectStore*, int, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, MonClient*, std::string const&amp;, std::string const&amp;)</fn>
      <dir>/usr/src/debug/ceph-12.0.3-2488-g7d2854b/src/osd</dir>
      <file>OSD.cc</file>
      <line>1948</line>
    </frame>
    <frame>
      <ip>0x502C04</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>main</fn>
      <dir>/usr/src/debug/ceph-12.0.3-2488-g7d2854b/src</dir>
      <file>ceph_osd.cc</file>
      <line>613</line>
    </frame>
  </stack>
</error>

/a/kchai-2017-07-04_04:14:45-rados-wip-20432-kefu-distro-basic-mira/1357991/remote/mira038/log/valgrind/osd.2.log.gz


Related issues

Duplicated by RADOS - Bug #39723: osd: valgrind Leak_DefinitelyLost Duplicate
Duplicated by RADOS - Bug #38403: osd: leaked from OSDMap::apply_incremental Duplicate 02/20/2019

History

#1 Updated by Kefu Chai almost 3 years ago

  • /a/kchai-2017-07-04_06:08:32-rados-wip-20432-kefu-distro-basic-mira/1359525/remote/mira038/log/valgrind/osd.0.log.gz
  • /a/kchai-2017-07-04_06:08:32-rados-wip-20432-kefu-distro-basic-mira/1359531/remote/mira061/log/valgrind/osd.5.log.gz

#2 Updated by Kefu Chai almost 3 years ago

  • Priority changed from Normal to High

#3 Updated by Sage Weil over 2 years ago

  • Status changed from New to Can't reproduce

#4 Updated by Kefu Chai over 1 year ago

  • Status changed from Can't reproduce to New
<error>
  <unique>0x58</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>9,549 (1,056 direct, 8,493 indirect) bytes in 1 blocks are definitely lost in loss record 89 of 92</text>
    <leakedbytes>9549</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA80F8E8</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new[](unsigned long)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.13.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>423</line>
    </frame>
    <frame>
      <ip>0x6B406D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSDService::OSDService(OSD*)</fn>
      <dir>/usr/src/debug/ceph-14.0.1-1422-g4cc9538/src/osdc</dir>
      <file>Objecter.h</file>
      <line>2020</line>
    </frame>
    <frame>
      <ip>0x6B769F</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::OSD(CephContext*, ObjectStore*, int, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, Messenger*, MonClient*, std::string const&amp;, std::string const&amp;)</fn>
      <dir>/usr/src/debug/ceph-14.0.1-1422-g4cc9538/src/osd</dir>
      <file>OSD.cc</file>
      <line>2116</line>
    </frame>
...

/a/kchai-2018-12-06_14:36:33-rados-wip-kefu2-testing-2018-12-06-1345-distro-basic-smithi/3311506/remote/smithi106/log/valgrind/osd.6.log.gz

#5 Updated by Sage Weil 11 months ago

  • Subject changed from objecter leaked in OSD to objecter leaked OSDMap in handle_osd_map
  • Status changed from New to In Progress
  • Assignee set to Sage Weil
    osdmap(new OSDMap),

in the ctor is the leaked line. I think teh bug is handle_osd_map, here:

    else if (m->maps.count(e)) {
      ldout(cct, 3) << "handle_osd_map decoding full epoch " << e << dendl;
          OSDMap *new_osdmap = new OSDMap();
          new_osdmap->decode(m->maps[e]);

          emit_blacklist_events(*osdmap, *new_osdmap);

          osdmap = new_osdmap;

#6 Updated by Sage Weil 11 months ago

  • Status changed from In Progress to Fix Under Review

https://github.com/ceph/ceph/pull/28242

I think we shouldn't backport the fix, as it might upset misbehaved (unlocked) callers, and this is a very low severity bug.

#7 Updated by Greg Farnum 10 months ago

  • Duplicated by Bug #39723: osd: valgrind Leak_DefinitelyLost added

#8 Updated by Sage Weil 10 months ago

  • Status changed from Fix Under Review to Resolved

#9 Updated by Sage Weil 9 months ago

  • Duplicated by Bug #38403: osd: leaked from OSDMap::apply_incremental added

Also available in: Atom PDF