Project

General

Profile

Actions
Copy link

Bug #15785

closed

valgrind osd leak Pipe::*

Added by Samuel Just almost 8 years ago. Updated over 7 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

<error>
<unique>0x63</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>4,705 (32 direct, 4,673 indirect) bytes in 1 blocks are definitely lost in loss record 100 of 102</text>
<leakedbytes>4705</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0xACCB0E0</ip>
<obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new(unsigned long)</fn>
</frame>
<frame>
<ip>0x4CAA53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>allocate</fn>
<dir>/usr/include/c++/4.8/ext</dir>
<file>new_allocator.h</file>
<line>104</line>
</frame>
<frame>
<ip>0x4CAA53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>_M_get_node</fn>
<dir>/usr/include/c++/4.8/bits</dir>
<file>stl_list.h</file>
<line>334</line>
</frame>
<frame>
<ip>0x4CAA53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>_M_create_node<ceph::buffer::ptr></fn>
<dir>/usr/include/c++/4.8/bits</dir>
<file>stl_list.h</file>
<line>502</line>
</frame>
<frame>
<ip>0x4CAA53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>_M_insert<ceph::buffer::ptr></fn>
<dir>/usr/include/c++/4.8/bits</dir>
<file>stl_list.h</file>
<line>1561</line>
</frame>
<frame>
<ip>0x4CAA53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::push_back(ceph::buffer::ptr&&)</fn>
<dir>/usr/include/c++/4.8/bits</dir>
<file>stl_list.h</file>
<line>1021</line>
</frame>
<frame>
<ip>0xBCFDE0</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>push_back</fn>
<dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.0-31-g4dbc24d/src/./include</dir>
<file>buffer.h</file>
<line>476</line>
</frame>
<frame>
<ip>0xBCFDE0</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
<dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.0-31-g4dbc24d/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>1976</line>
</frame>
<frame>
<ip>0xBDCCC3</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::reader()</fn>
<dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.0-31-g4dbc24d/src/msg/simple</dir>
<file>Pipe.cc</file>
<line>1586</line>
</frame>
<frame>
<ip>0xBE572C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>Pipe::Reader::entry()</fn>
<dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.0-31-g4dbc24d/src/msg/simple</dir>
<file>Pipe.h</file>
<line>52</line>
</frame>
<frame>
<ip>0xBB98181</ip>
<obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
<fn>start_thread</fn>
<dir>/build/eglibc-3GlaMS/eglibc-2.19/nptl</dir>
<file>pthread_create.c</file>
<line>312</line>
</frame>
<frame>
<ip>0xDAA747C</ip>
<obj>/lib/x86_64-linux-gnu/libc-2.19.so</obj>
<fn>clone</fn>
<dir>/build/eglibc-3GlaMS/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64</dir>
<file>clone.S</file>
<line>111</line>

sjust@teuthology:/a/yehudasa-2016-05-08_09:42:38-rgw-wip-rgw-leak-3---basic-smithi/179556/remote

Related issues 1 (0 open — 1 closed)

Related to Messengers - Bug #15517: msg/simple: PipeConnection leakCan't reproduce04/15/2016

Actions
Actions
Copy link
 #1

Updated by Samuel Just almost 8 years ago

  • Project changed from rgw to Ceph
Actions
Copy link
 #2

Updated by Samuel Just almost 8 years ago

sjust@teuthology:/a/yehudasa-2016-05-11_11:03:48-rgw-wip-rgw-leak-4---basic-smithi/185152/remote

Actions
Copy link
 #3

Updated by Loïc Dachary over 7 years ago

  • Subject changed from valgrind osd leak to valgrind osd leak Pipe::read_message
Actions
Copy link
 #4

Updated by Loïc Dachary over 7 years ago 

  <unique>0x5a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,022 (360 direct, 662 indirect) bytes in 1 blocks are definitely lost in loss record 91 of 99</text>
    <leakedbytes>1022</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xACD10E0</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0xBD6829</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Pipe(SimpleMessenger*, int, PipeConnection*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>95</line>
    </frame>
    <frame>
      <ip>0xAC4DAC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>SimpleMessenger::add_accept_pipe(int)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>SimpleMessenger.cc</file>
      <line>343</line>
    </frame>
    <frame>
      <ip>0xB88C7A</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Accepter::entry()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Accepter.cc</file>
      <line>246</line>
    </frame>
    <frame>
      <ip>0xBB9E183</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <fn>start_thread</fn>
      <dir>/build/eglibc-oGUzwX/eglibc-2.19/nptl</dir>
      <file>pthread_create.c</file>
      <line>312</line>
    </frame>
    <frame>
      <ip>0xDAAD37C</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.19.so</obj>
      <fn>clone</fn>
      <dir>/build/eglibc-oGUzwX/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64</dir>
      <file>clone.S</file>
      <line>111</line>
    </frame>
  </stack>

  <unique>0x63</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>2,009 (560 direct, 1,449 indirect) bytes in 1 blocks are definitely lost in loss record 100 of 102</text>
    <leakedbytes>2009</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xACD10E0</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/build/valgrind-hP3I3v/valgrind-3.10.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>324</line>
    </frame>
    <frame>
      <ip>0xA43C19</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>decode_message(CephContext*, int, ceph_msg_header&amp;, ceph_msg_footer&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg</dir>
      <file>Message.cc</file>
      <line>452</line>
    </frame>
    <frame>
      <ip>0xBD3DFC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>2082</line>
    </frame>
    <frame>
      <ip>0xBE0AE3</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1594</line>
    </frame>
    <frame>
      <ip>0xBE93AC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>52</line>
    </frame>
    <frame>
      <ip>0xBB9E181</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <fn>start_thread</fn>
      <dir>/build/buildd/eglibc-2.19/nptl</dir>
      <file>pthread_create.c</file>
      <line>312</line>
    </frame>
    <frame>
      <ip>0xDAAD47C</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.19.so</obj>
      <fn>clone</fn>
      <dir>/build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64</dir>
      <file>clone.S</file>
      <line>111</line>
    </frame>
  </stack>

  <unique>0x5a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,022 (408 direct, 614 indirect) bytes in 1 blocks are definitely lost in loss record 90 of 99</text>
    <leakedbytes>1022</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xAD06105</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x46EF69</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&amp;, ceph::buffer::list&amp;, bool&amp;, CryptoKey&amp;)</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/osd</dir>
      <file>OSD.cc</file>
      <line>5959</line>
    </frame>
    <frame>
      <ip>0xAF1C3E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ms_deliver_verify_authorizer</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg</dir>
      <file>Messenger.h</file>
      <line>711</line>
    </frame>
    <frame>
      <ip>0xAF1C3E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>SimpleMessenger::verify_authorizer(Connection*, int, int, ceph::buffer::list&amp;, ceph::buffer::list&amp;, bool&amp;, CryptoKey&amp;)</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>SimpleMessenger.cc</file>
      <line>398</line>
    </frame>
    <frame>
      <ip>0xC1410D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::accept()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>442</line>
    </frame>
    <frame>
      <ip>0xC1941E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1507</line>
    </frame>
    <frame>
      <ip>0xC20F2C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>52</line>
    </frame>
    <frame>
      <ip>0xC13BDC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xDAE828C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>

It does not always happen. http://pulpito.ceph.com/loic-2016-08-10_23:15:44-rados-jewel---basic-smithi/ has 10 runs of the same job rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml} and only 3 of them failed.

Actions
Copy link
 #5

Updated by Loïc Dachary over 7 years ago

  • Subject changed from valgrind osd leak Pipe::read_message to valgrind osd leak Pipe::*
Actions
Copy link
 #6

Updated by Loïc Dachary over 7 years ago

  • Related to Bug #15517: msg/simple: PipeConnection leak added
Actions
Copy link
 #7

Updated by Samuel Just over 7 years ago

  • Assignee set to Samuel Just
Actions
Copy link
 #8

Updated by Samuel Just over 7 years ago

  • Assignee deleted (Samuel Just)
Actions
Copy link
 #9

Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF