https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-04-11T19:48:51ZCeph bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1109292018-04-11T19:48:51ZSage Weilsage@newdream.net
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>bluestore</i></li><li><strong>Subject</strong> changed from <i>"Caught signal" in smoke on rhel7.5</i> to <i>tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_db</i></li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1111802018-04-13T18:38:10ZSage Weilsage@newdream.net
<ul></ul><p>This looks to me like a build issue with tcmalloc... specifically, building in centos and running in rhel. Running on rhel with a notmcalloc build is fine.</p>
<p>Also, notably, simply running ceph-mon with no arguments, which exits out of main() before doing almost anything at all, results in the tcmalloc message and segfault. This suggests that something is happening in a static singleton definition that is going wrong. My guess is a rocksdb singleton that (incorrect) does something with tcmalloc and fails due to an incompatible ABI?</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1111812018-04-13T18:45:38ZSage Weilsage@newdream.net
<ul></ul><p>on lab centos deploy,<br /><pre>
[sage@smithi099 ~]$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.4.1708 (Core)
Release: 7.4.1708
Codename: Core
[sage@smithi099 ~]$ rpm -qf /lib64/libtcmalloc.so.4
gperftools-libs-2.4-8.el7.x86_64
</pre><br />vs rhel<br /><pre>
[sage@smithi095 ~]$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.4 (Maipo)
Release: 7.4
Codename: Maipo
[sage@smithi095 ~]$ rpm -qf /lib64/libtcmalloc.so.4
gperftools-libs-2.4-8.el7.x86_64
</pre></p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1111822018-04-13T18:46:40ZSage Weilsage@newdream.net
<ul></ul><p>except the job runs on rhel 7.5,<br /><pre>
[sage@smithi116 ~]$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.5 (Maipo)
Release: 7.5
Codename: Maipo
[sage@smithi116 ~]$ rpm -qf /lib64/libtcmalloc.so.4
gperftools-libs-2.6.1-1.el7.x86_64
</pre><br />but there is no centos 7.5 image.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1114822018-04-18T22:01:50ZSage Weilsage@newdream.net
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Immediate</i></li></ul><p>/a/sage-2018-04-18_19:08:00-rados-wip-sage-testing-2018-04-18-1210-distro-basic-smithi/2413082</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1115022018-04-19T07:57:44ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Duplicate</i></li></ul><p>tcmalloc 2.6.1 is buggy. probably we need runtime check to disallow ceph to run with "2.5 < tcmalloc.version < 2.6.2" in debian/control or ceph.spec.</p>
<p>or better off, applying the patch in <a class="external" href="https://bugzilla.redhat.com/show_bug.cgi?id=1494309">https://bugzilla.redhat.com/show_bug.cgi?id=1494309</a> to gperftools-libs-2.6.1-1.el7.x86_64 shipped with RHEL 7.5</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1115042018-04-19T08:12:12ZKefu Chaitchaikov@gmail.com
<ul></ul><p>i just filed <a class="external" href="https://bugzilla.redhat.com/show_bug.cgi?id=1569391">https://bugzilla.redhat.com/show_bug.cgi?id=1569391</a> to track this issue at downstream.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1115052018-04-19T08:14:00ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed" href="/issues/21422">Bug #21422</a>: crash in rocksdb LRUCache destructor with tcmalloc v4.2.6 / gperf-tools v2.5.93</i> added</li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1115072018-04-19T08:14:57ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Duplicate</i> to <i>12</i></li></ul><p>change the status to verified. because, unlike <a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: crash in rocksdb LRUCache destructor with tcmalloc v4.2.6 / gperf-tools v2.5.93 (Resolved)" href="https://tracker.ceph.com/issues/21422">#21422</a>, this issue more of a run-time dependency problem.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1116902018-04-24T06:44:08ZKefu Chaitchaikov@gmail.com
<ul></ul><p>i was thinking about statically linking against tcmalloc, but seems it's a dead-end.</p>
<p>see <a class="external" href="https://sourceware.org/bugzilla/show_bug.cgi?id=20432">https://sourceware.org/bugzilla/show_bug.cgi?id=20432</a>. and the glibc bug was fixed in 2.25, but RHEL/centos 7.4 comes with glibc v2.17. so we cannot link tcmalloc statically on RHEL/centos safely.</p>
<p>currently ceph pulls the gperftools-libs by depending on libtcmalloc.so.4. we could "Requires" gperftools-libs explicitly. but rpm's spec does not allow something like<br /><pre>
Requires: gperftools-libs != 2.6.1-1
</pre></p>
<p>because <= 2.5, and >= 2.6.1-5 do not have this issue.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1116912018-04-24T06:55:05ZKefu Chaitchaikov@gmail.com
<ul></ul><p>or we can notes this down as a known issue on RHEL7.5 and gperftools-libs 2.6.1-1.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1117232018-04-24T15:30:03ZJosh Durgin
<ul><li><strong>Assignee</strong> set to <i>Kefu Chai</i></li></ul><p>Discussed on irc, it appears we can work around this by replacing the single aligned_alloc() call in rocksdb with posix_memalign(), which we already use in bufferlist.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1117672018-04-25T03:59:50ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Fix Under Review</i></li></ul><p>- <a class="external" href="https://github.com/ceph/ceph/pull/21632">https://github.com/ceph/ceph/pull/21632</a><br />- <a class="external" href="https://github.com/ceph/rocksdb/pull/36">https://github.com/ceph/rocksdb/pull/36</a></p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1118032018-04-25T12:49:44ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1132202018-05-15T12:58:50ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>12</i></li></ul><p>we are now using centos 7.5 for building rpm. so we should drop this change in cmake.<br /><pre>
Getting requirements for /tmp/install-deps.11183/ceph.spec
--> 1:java-1.8.0-openjdk-devel-1.8.0.171-7.b10.el7.x86_64
--> sharutils-4.13.3-8.el7.x86_64
--> Already installed : checkpolicy-2.5-4.el7.x86_64
--> selinux-policy-devel-3.13.1-192.el7_5.3.noarch
--> Already installed : bc-1.06.95-13.el7.x86_64
--> gperf-3.0.4-8.el7.x86_64
--> Already installed : cmake-2.8.12.2-2.el7.x86_64
--> cryptsetup-1.7.4-4.el7.x86_64
--> fuse-devel-2.9.2-10.el7.x86_64
--> devtoolset-7-gcc-c++-7.2.1-1.el7.sc1.x86_64
--> Already installed : gdbm-1.10-8.el7.x86_64
--> gperftools-devel-2.6.1-1.el7.x86_64
</pre></p>
<pre>
CMake Error at cmake/modules/BuildRocksDB.cmake:64 (message):
Incompatible tcmalloc v2.6.1 and rocksdb v5.13.0, please install
gperf-tools 2.5 (not 2.5.93) or >= 2.6.2, or switch to another allocator
using 'cmake -DALLOCATOR=libc'.
Call Stack (most recent call first):
cmake/modules/BuildRocksDB.cmake:94 (check_aligned_alloc)
src/CMakeLists.txt:860 (build_rocksdb)
</pre> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1133732018-05-17T02:51:48ZKefu Chaitchaikov@gmail.com
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/22046">https://github.com/ceph/ceph/pull/22046</a> to drop the check for tcmalloc</p>
<p><a class="external" href="https://github.com/facebook/rocksdb/pull/3862">https://github.com/facebook/rocksdb/pull/3862</a> is posted to address the issue on rocksdb side.</p> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1133742018-05-17T03:04:23ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Fix Under Review</i></li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1133772018-05-17T06:05:07ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1133782018-05-17T06:09:16ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-7 priority-highest closed" href="/issues/24154">Backport #24154</a>: mimic: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_db</i> added</li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1136522018-05-21T11:46:06ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Backport</strong> set to <i>mimic</i></li></ul> bluestore - Bug #23653: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbhttps://tracker.ceph.com/issues/23653?journal_id=1206502018-09-13T11:00:31ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed" href="/issues/35969">Bug #35969</a>: "symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm" on centos 7.4</i> added</li></ul>