Ceph » bluestore

I always got this segmentation fault when I use jemalloc. If use the default tcmalloc, it is fine now.

I believe the problem here is that the rocksdb is more tightly bound to tcmalloc. Not exactly sure what the issues or fix is.

Hi, this happens with RockDB anf filestore on luminous 12.2 on ubuntu 16.04.
Commenting /etc/default/ceph line LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 and this is still happening.

any news ?

Hi Mikko,
What architecture are you running on?
I tried to match your callstacks with binaries for x86_64 for "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)" and it does not match. I took binaries from https://download.ceph.com/rpm-luminous/el7/x86_64/ .
Did you compile yourself?
Best regards,
Adam

The arch is x68_64. Ceph was installed from eu.ceph.com deb repo. This issue isn't current for me anymore as the cluster has been upgraded to newest release in the meantime and I've given up on jemalloc. After going back to tcmalloc, the issue hasn't resurfaced.

Hi, just wanted to report I'm hitting the same issue on centos 7 with jemalloc-3.6.0-1.el7 and ceph 12.2.2

Got a similar issue on Jewel 10.2.11 as it uses rocksdb by default now. I know Jewel is near EOL and the plan was to update to latest Jewel and then upgrade to Luminous but this pus everything on hold.

I've disabled jemalloc for the OSDs that kept crashing and everything seems fine for now.

OS: CentOS 7
Ceph packages: http://download.ceph.com/rpm-jewel/el7/x86_64/
jemalloc: 3.6.0-1 from EPEL

 ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
 1: (()+0x9f1c2a) [0x7f9e2c4ebc2a]
 2: (()+0xf5e0) [0x7f9e29fd65e0]
 3: (()+0x34484) [0x7f9e2b6b4484]
 4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xef) [0x7f9e2c3eeb0f]
 5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x427) [0x7f9e2c3efc67]
 6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x136) [0x7f9e2c3efee6]
 7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x98) [0x7f9e2c3f8848]
 8: (()+0x92268e) [0x7f9e2c41c68e]
 9: (()+0x922cad) [0x7f9e2c41ccad]
 10: (()+0x922cbf) [0x7f9e2c41ccbf]
 11: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xde) [0x7f9e2c403d1e]
 12: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x189) [0x7f9e2c4827f9]
 13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0x45) [0x7f9e2c36a1a5]
 14: (DBObjectMap::DBObjectMapIteratorImpl::lower_bound(std::string const&)+0x47) [0x7f9e2c3182c7]
 15: (FileStore::_omap_rmkeyrange(coll_t const&, ghobject_t const&, std::string const&, std::string const&, SequencerPosition const&)+0x217) [0x7f9e2c2467f7]
 16: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x10f7) [0x7f9e2c267a27]
 17: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, unsigned long, ThreadPool::TPHandle*)+0x3b) [0x7f9e2c26d56b]
 18: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x2cd) [0x7f9e2c26d86d]
 19: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x7f9e2c5dbbae]
 20: (ThreadPool::WorkThread::entry()+0x10) [0x7f9e2c5dca90]
 21: (()+0x7e25) [0x7f9e29fcee25]
 22: (clone()+0x6d) [0x7f9e2865934d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.