Project

General

Profile

Bug #15356

Objecter: valgrind unclean and suppression not working in CentOS

Added by Greg Farnum almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
qa
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/teuthology-2016-03-28_18:04:01-fs-master---basic-smithi/92567/

This was fixed in #14794 for our Ubuntu boxes, but for some reason it's not working in the CentOS ones.

Looks like our unique_lock either has an issue or the boost bit needs a valgrind exception.

<error>
  <unique>0xd</unique>
  <tid>1</tid>
  <kind>Leak_StillReachable</kind>
  <xwhat>
    <text>8 bytes in 1 blocks are still reachable in loss record 13 of 66</text>
    <leakedbytes>8</leakedbytes>
    <leakedblocks>1</leakedblocks>
  </xwhat>
  <stack>
    <frame>
      <ip>0xA564BFD</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>malloc</fn>
    </frame>
    <frame>
      <ip>0xA9BE7F9</ip>
      <obj>/usr/lib64/libboost_thread-mt.so.1.53.0</obj>
      <fn>boost::detail::get_once_per_thread_epoch()</fn>
    </frame>
    <frame>
      <ip>0x4CF647</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>void boost::call_once&lt;void (*)()&gt;(boost::once_flag&amp;, void (*)())</fn>
    </frame>
    <frame>
      <ip>0xA9B8C16</ip>
      <obj>/usr/lib64/libboost_thread-mt.so.1.53.0</obj>
      <fn>boost::detail::get_current_thread_data()</fn>
    </frame>
    <frame>
      <ip>0xA9B8C38</ip>
      <obj>/usr/lib64/libboost_thread-mt.so.1.53.0</obj>
      <fn>boost::this_thread::interruption_enabled()</fn>
    </frame>
    <frame>
      <ip>0xA9B8C68</ip>
      <obj>/usr/lib64/libboost_thread-mt.so.1.53.0</obj>
      <fn>boost::this_thread::disable_interruption::disable_interruption()</fn>
    </frame>
    <frame>
      <ip>0x384A1E</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>boost::shared_mutex::lock()</fn>
    </frame>
    <frame>
      <ip>0x35F3EF</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>Objecter::update_crush_location()</fn>
    </frame>
    <frame>
      <ip>0x35FA6E</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>Objecter::init()</fn>
    </frame>
    <frame>
      <ip>0x2AF74A</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>Client::init()</fn>
    </frame>
    <frame>
      <ip>0x2909E3</ip>
      <obj>/usr/bin/ceph-fuse</obj>
      <fn>main</fn>
    </frame>
  </stack>
</error>

Related issues

Copied from Ceph - Bug #14794: Objecter: valgrind unclean Resolved 02/18/2016

History

#1 Updated by Greg Farnum almost 8 years ago

  • Copied from Bug #14794: Objecter: valgrind unclean added

#2 Updated by Greg Farnum almost 8 years ago

  • Assignee deleted (Adam Emerson)

#3 Updated by Greg Farnum almost 8 years ago

  • Priority changed from Normal to High

John Spray wrote:

We're still seeing this regularly: the suppression is in there but for some reason it isn't taking effect. Today's instance: http://pulpito.ceph.com/teuthology-2016-05-21_17:15:02-fs-master---basic-smithi/206167/

#4 Updated by John Spray almost 8 years ago

Tried tweaking the suppression like this as an experiment, but no joy.

-       fun:boost::detail::get_once_per_thread_epoch()
-       fun:void boost::call_once*
-       fun:boost::detail::get_current_thread_data()
+       fun:boost::detail::get_once_per_thread_epoch
+       fun:boost::call_once*
+       fun:boost::detail::get_current_thread_data

Trying just filtering by obj: instead of fun: now...

#5 Updated by John Spray over 7 years ago

The last thing I tried (which also didn't help) was this:

diff --git a/teuthology/task/valgrind.supp b/teuthology/task/valgrind.supp
index 201339e..1745370 100644
--- a/teuthology/task/valgrind.supp
+++ b/teuthology/task/valgrind.supp
@@ -302,11 +302,9 @@
 {
        thread_local memory is falsely detected (https://svn.boost.org/trac/boost/ticket/3296)
        Memcheck:Leak
-       ...
-       fun:boost::detail::get_once_per_thread_epoch()
-       fun:void boost::call_once*
-       fun:boost::detail::get_current_thread_data()
-       ...
+    ...
+    obj:/usr/lib64/libboost_thread-mt.*
+    ...
 }
 {
        rocksdb thread local singletons

#6 Updated by John Spray over 7 years ago

I'm pretty much out of ideas, so I've written to valgrind-users about this to see if anyone can help

#7 Updated by Sage Weil over 7 years ago

Are you still seeing this? I thought we had the thread singletons (finally) whitelisted now..

#8 Updated by John Spray over 7 years ago

  • Status changed from New to Resolved

iirc this was fixed by https://github.com/ceph/teuthology/pull/923, we're not seeing it any more anyway

Also available in: Atom PDF