Ceph

sjust@teuthology:/a/yehudasa-2016-05-11_11:03:48-rgw-wip-rgw-leak-4---basic-smithi/185152/remote

• http://pulpito.ceph.com/loic-2016-08-10_23:15:44-rados-jewel---basic-smithi/358138/

  <unique>0x5a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,022 (360 direct, 662 indirect) bytes in 1 blocks are definitely lost in loss record 91 of 99</text>
    <leakedbytes>1022</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xACD10E0</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0xBD6829</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Pipe(SimpleMessenger*, int, PipeConnection*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>95</line>
    </frame>
    <frame>
      <ip>0xAC4DAC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>SimpleMessenger::add_accept_pipe(int)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>SimpleMessenger.cc</file>
      <line>343</line>
    </frame>
    <frame>
      <ip>0xB88C7A</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Accepter::entry()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Accepter.cc</file>
      <line>246</line>
    </frame>
    <frame>
      <ip>0xBB9E183</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <fn>start_thread</fn>
      <dir>/build/eglibc-oGUzwX/eglibc-2.19/nptl</dir>
      <file>pthread_create.c</file>
      <line>312</line>
    </frame>
    <frame>
      <ip>0xDAAD37C</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.19.so</obj>
      <fn>clone</fn>
      <dir>/build/eglibc-oGUzwX/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64</dir>
      <file>clone.S</file>
      <line>111</line>
    </frame>
  </stack>

• http://pulpito.ceph.com/loic-2016-08-10_23:15:44-rados-jewel---basic-smithi/358135/

  <unique>0x63</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>2,009 (560 direct, 1,449 indirect) bytes in 1 blocks are definitely lost in loss record 100 of 102</text>
    <leakedbytes>2009</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xACD10E0</ip>
      <obj>/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
      <dir>/build/valgrind-hP3I3v/valgrind-3.10.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>324</line>
    </frame>
    <frame>
      <ip>0xA43C19</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>decode_message(CephContext*, int, ceph_msg_header&amp;, ceph_msg_footer&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;, ceph::buffer::list&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg</dir>
      <file>Message.cc</file>
      <line>452</line>
    </frame>
    <frame>
      <ip>0xBD3DFC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::read_message(Message**, AuthSessionHandler*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>2082</line>
    </frame>
    <frame>
      <ip>0xBE0AE3</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1594</line>
    </frame>
    <frame>
      <ip>0xBE93AC</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-10.2.2-239-gc587e9e/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>52</line>
    </frame>
    <frame>
      <ip>0xBB9E181</ip>
      <obj>/lib/x86_64-linux-gnu/libpthread-2.19.so</obj>
      <fn>start_thread</fn>
      <dir>/build/buildd/eglibc-2.19/nptl</dir>
      <file>pthread_create.c</file>
      <line>312</line>
    </frame>
    <frame>
      <ip>0xDAAD47C</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.19.so</obj>
      <fn>clone</fn>
      <dir>/build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64</dir>
      <file>clone.S</file>
      <line>111</line>
    </frame>
  </stack>

• http://pulpito.ceph.com/loic-2016-08-10_23:15:44-rados-jewel---basic-smithi/358136/

  <unique>0x5a</unique>
  <tid>1</tid>
  <kind>Leak_DefinitelyLost</kind>
  <xwhat>
    <text>1,022 (408 direct, 614 indirect) bytes in 1 blocks are definitely lost in loss record 90 of 99</text>
    <leakedbytes>1022</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xAD06105</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long)</fn>
    </frame>
    <frame>
      <ip>0x46EF69</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>OSD::ms_verify_authorizer(Connection*, int, int, ceph::buffer::list&amp;, ceph::buffer::list&amp;, bool&amp;, CryptoKey&amp;)</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/osd</dir>
      <file>OSD.cc</file>
      <line>5959</line>
    </frame>
    <frame>
      <ip>0xAF1C3E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>ms_deliver_verify_authorizer</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg</dir>
      <file>Messenger.h</file>
      <line>711</line>
    </frame>
    <frame>
      <ip>0xAF1C3E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>SimpleMessenger::verify_authorizer(Connection*, int, int, ceph::buffer::list&amp;, ceph::buffer::list&amp;, bool&amp;, CryptoKey&amp;)</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>SimpleMessenger.cc</file>
      <line>398</line>
    </frame>
    <frame>
      <ip>0xC1410D</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::accept()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>442</line>
    </frame>
    <frame>
      <ip>0xC1941E</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::reader()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.cc</file>
      <line>1507</line>
    </frame>
    <frame>
      <ip>0xC20F2C</ip>
      <obj>/usr/bin/ceph-osd</obj>
      <fn>Pipe::Reader::entry()</fn>
      <dir>/usr/src/debug/ceph-10.2.2/src/msg/simple</dir>
      <file>Pipe.h</file>
      <line>52</line>
    </frame>
    <frame>
      <ip>0xC13BDC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0xDAE828C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>

It does not always happen. http://pulpito.ceph.com/loic-2016-08-10_23:15:44-rados-jewel---basic-smithi/ has 10 runs of the same job rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml} and only 3 of them failed.