Project

General

Profile

Bug #21295

OSD Seg Fault on Bluestore OSD

Added by Barry Stokes about 2 years ago. Updated almost 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
09/07/2017
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

I have a Bluestore OSDs which keeps failing with a Seg Fault:

2017-09-07 12:18:27.749364 7fa21813be40 -1 ** Caught signal (Segmentation fault) *
in thread 7fa21813be40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55611f89f2f4]
2: (()+0x11670) [0x7fa2163e4670]
3: (()+0x1fd9f) [0x7fa217d17d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55611fc6efbc]
5: (rocksdb::BlockBasedTable::NewIterator(rocksdb::ReadOptions const&, rocksdb::Arena*, bool)+0x26) [0x55611fc760d6]
6: (rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::EnvOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::RangeDelAggregator*, rocksdb::TableReader**, rocksdb::Histogra
mImpl*, bool, rocksdb::Arena*, bool, int)+0x531) [0x55611fd536f1]
7: (()+0xdd7f25) [0x55611fc20f25]
8: (()+0xe5849e) [0x55611fca149e]
9: (()+0xe58a16) [0x55611fca1a16]
10: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0x126) [0x55611fc82b76]
11: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x20a) [0x55611fd22e9a]
12: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&
)+0x46) [0x55611f7e2076]
13: (BlueStore::OmapIteratorImpl::OmapIteratorImpl(boost::intrusive_ptr<BlueStore::Collection>, boost::intrusive_ptr<BlueStore::Onode>, std::shared_ptr<KeyValueDB::IteratorImpl>)+0x154) [0x55611f74bf74]
14: (BlueStore::get_omap_iterator(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&)+0x292) [0x55611f76bcc2]
15: (BlueStore::get_omap_iterator(coll_t const&, ghobject_t const&)+0x5d) [0x55611f75afdd]
16: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::__cxx11::basic_ostringstream<char, std::char_traits<char>, st
d::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
ar> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+0x162) [0x55611f403792]
17: (PG::read_state(ObjectStore
, ceph::buffer::list&)+0x399) [0x55611f3b1819]
18: (OSD::load_pgs()+0x8cd) [0x55611f3043ad]
19: (OSD::init()+0x2217) [0x55611f31f187]
20: (main()+0x2eb8) [0x55611f22f538]
21: (__libc_start_main()+0xf1) [0x7fa2153643f1]
22: (_start()+0x29) [0x55611f2bae69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

It comes back up for a time but then fails again:

2017-09-07 12:14:08.949740 7f59087e4700 -1 ** Caught signal (Segmentation fault) *
in thread 7f59087e4700 thread_name:tp_osd_tp

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55f739db62f4]
2: (()+0x11670) [0x7f5923e09670]
3: (()+0x1fd9f) [0x7f592573cd9f]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0x1d9) [0x55f73a182d39]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x3b7) [0x55f73a184717]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x55f73a184a5c]
7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x55f73a18d1d7]
8: (()+0xe5849e) [0x55f73a1b849e]
9: (()+0xe58a16) [0x55f73a1b8a16]
10: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0x126) [0x55f73a199b76]
11: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x20a) [0x55f73a239e9a]
12: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x46) [0x55f739cf9076]
13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x29) [0x55f739cf8a89]
14: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t)+0x97b) [0x55f739c54c9b]
15: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t)+0x25a) [0x55f739c5692a]
16: (PGBackend::objects_list_partial(hobject_t const&, int, int, std::vector<hobject_t, std::allocator<hobject_t> >, hobject_t)+0x3a0) [0x55f739a17790]
17: (PrimaryLogPG::scan_range(int, int, PG::BackfillInterval*, ThreadPool::TPHandle&)+0x1d3) [0x55f739976633]
18: (PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x820) [0x55f739977ab0]
19: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x671) [0x55f739981ed1]
20: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x55f7398066b9]
21: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x55f739a9df57]
22: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x55f73982de3e]
23: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x55f739dfde64]
24: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55f739e00ea0]
25: (()+0x76da) [0x7f5923dff6da]
26: (clone()+0x5f) [0x7f5922e71d7f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 12:14:08.949740 7f59087e4700 -1 ** Caught signal (Segmentation fault) *
in thread 7f59087e4700 thread_name:tp_osd_tp

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55f739db62f4]
2: (()+0x11670) [0x7f5923e09670]
3: (()+0x1fd9f) [0x7f592573cd9f]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0x1d9) [0x55f73a182d39]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x3b7) [0x55f73a184717]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x55f73a184a5c]
7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x55f73a18d1d7]
8: (()+0xe5849e) [0x55f73a1b849e]
9: (()+0xe58a16) [0x55f73a1b8a16]
10: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0x126) [0x55f73a199b76]
11: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x20a) [0x55f73a239e9a]
12: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x46) [0x55f739cf9076]
13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x29) [0x55f739cf8a89]
14: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t)+0x97b) [0x55f739c54c9b]
15: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t)+0x25a) [0x55f739c5692a]
16: (PGBackend::objects_list_partial(hobject_t const&, int, int, std::vector<hobject_t, std::allocator<hobject_t> >, hobject_t)+0x3a0) [0x55f739a17790]
17: (PrimaryLogPG::scan_range(int, int, PG::BackfillInterval*, ThreadPool::TPHandle&)+0x1d3) [0x55f739976633]
18: (PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x820) [0x55f739977ab0]
19: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x671) [0x55f739981ed1]
20: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x55f7398066b9]
21: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x55f739a9df57]
22: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x55f73982de3e]
23: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x55f739dfde64]
24: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55f739e00ea0]
25: (()+0x76da) [0x7f5923dff6da]
26: (clone()+0x5f) [0x7f5922e71d7f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 12:18:27.749364 7fa21813be40 -1 ** Caught signal (Segmentation fault) *
in thread 7fa21813be40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55611f89f2f4]
2: (()+0x11670) [0x7fa2163e4670]
3: (()+0x1fd9f) [0x7fa217d17d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55611fc6efbc]
5: (rocksdb::BlockBasedTable::NewIterator(rocksdb::ReadOptions const&, rocksdb::Arena*, bool)+0x26) [0x55611fc760d6]
6: (rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::EnvOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::RangeDelAggregator*, rocksdb::TableReader**, rocksdb::HistogramImpl*, bool, rocksdb::Arena*, bool, int)+0x531) [0x55611fd536f1]
7: (()+0xdd7f25) [0x55611fc20f25]
8: (()+0xe5849e) [0x55611fca149e]
9: (()+0xe58a16) [0x55611fca1a16]
10: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0x126) [0x55611fc82b76]
11: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x20a) [0x55611fd22e9a]
12: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x46) [0x55611f7e2076]
13: (BlueStore::OmapIteratorImpl::OmapIteratorImpl(boost::intrusive_ptr<BlueStore::Collection>, boost::intrusive_ptr<BlueStore::Onode>, std::shared_ptr<KeyValueDB::IteratorImpl>)+0x154) [0x55611f74bf74]
14: (BlueStore::get_omap_iterator(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&)+0x292) [0x55611f76bcc2]
15: (BlueStore::get_omap_iterator(coll_t const&, ghobject_t const&)+0x5d) [0x55611f75afdd]
16: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+0x162) [0x55611f403792]
17: (PG::read_state(ObjectStore
, ceph::buffer::list&)+0x399) [0x55611f3b1819]
18: (OSD::load_pgs()+0x8cd) [0x55611f3043ad]
19: (OSD::init()+0x2217) [0x55611f31f187]
20: (main()+0x2eb8) [0x55611f22f538]
21: (__libc_start_main()+0xf1) [0x7fa2153643f1]
22: (_start()+0x29) [0x55611f2bae69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 12:18:27.749364 7fa21813be40 -1 ** Caught signal (Segmentation fault) *
in thread 7fa21813be40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55611f89f2f4]
2: (()+0x11670) [0x7fa2163e4670]
3: (()+0x1fd9f) [0x7fa217d17d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55611fc6efbc]
5: (rocksdb::BlockBasedTable::NewIterator(rocksdb::ReadOptions const&, rocksdb::Arena*, bool)+0x26) [0x55611fc760d6]
6: (rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::EnvOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::RangeDelAggregator*, rocksdb::TableReader**, rocksdb::HistogramImpl*, bool, rocksdb::Arena*, bool, int)+0x531) [0x55611fd536f1]
7: (()+0xdd7f25) [0x55611fc20f25]
8: (()+0xe5849e) [0x55611fca149e]
9: (()+0xe58a16) [0x55611fca1a16]
10: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0x126) [0x55611fc82b76]
11: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x20a) [0x55611fd22e9a]
12: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x46) [0x55611f7e2076]
13: (BlueStore::OmapIteratorImpl::OmapIteratorImpl(boost::intrusive_ptr<BlueStore::Collection>, boost::intrusive_ptr<BlueStore::Onode>, std::shared_ptr<KeyValueDB::IteratorImpl>)+0x154) [0x55611f74bf74]
14: (BlueStore::get_omap_iterator(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&)+0x292) [0x55611f76bcc2]
15: (BlueStore::get_omap_iterator(coll_t const&, ghobject_t const&)+0x5d) [0x55611f75afdd]
16: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+0x162) [0x55611f403792]
17: (PG::read_state(ObjectStore
, ceph::buffer::list&)+0x399) [0x55611f3b1819]
18: (OSD::load_pgs()+0x8cd) [0x55611f3043ad]
19: (OSD::init()+0x2217) [0x55611f31f187]
20: (main()+0x2eb8) [0x55611f22f538]
21: (__libc_start_main()+0xf1) [0x7fa2153643f1]
22: (_start()+0x29) [0x55611f2bae69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 12:24:18.406925 7fef749c8e40 -1 ** Caught signal (Segmentation fault) *
in thread 7fef749c8e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55bdadc842f4]
2: (()+0x11670) [0x7fef72c71670]
3: (()+0x1fd9f) [0x7fef745a4d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55bdae053fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x55bdae054d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x55bdae139275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x55bdae0148fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x55bdae0de30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x55bdae0de772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x55bdadbcf5a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list
)+0x144) [0x55bdadbc6cf4]
12: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0x52d) [0x55bdadb4faad]
13: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x55bdadb3fe5c]
14: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x55bdad772d78]
15: (OSD::load_pgs()+0x4a0) [0x55bdad6e8f80]
16: (OSD::init()+0x2217) [0x55bdad704187]
17: (main()+0x2eb8) [0x55bdad614538]
18: (__libc_start_main()+0xf1) [0x7fef71bf13f1]
19: (_start()+0x29) [0x55bdad69fe69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 12:24:18.406925 7fef749c8e40 -1 ** Caught signal (Segmentation fault) *
in thread 7fef749c8e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55bdadc842f4]
2: (()+0x11670) [0x7fef72c71670]
3: (()+0x1fd9f) [0x7fef745a4d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55bdae053fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x55bdae054d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x55bdae139275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x55bdae0148fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x55bdae0de30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x55bdae0de772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x55bdadbcf5a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list
)+0x144) [0x55bdadbc6cf4]
12: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0x52d) [0x55bdadb4faad]
13: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x55bdadb3fe5c]
14: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x55bdad772d78]
15: (OSD::load_pgs()+0x4a0) [0x55bdad6e8f80]
16: (OSD::init()+0x2217) [0x55bdad704187]
17: (main()+0x2eb8) [0x55bdad614538]
18: (__libc_start_main()+0xf1) [0x7fef71bf13f1]
19: (_start()+0x29) [0x55bdad69fe69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 12:28:17.575591 7f8a7c391e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f8a7c391e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55c8526752f4]
2: (()+0x11670) [0x7f8a7a63a670]
3: (()+0x1fd9f) [0x7f8a7bf6dd9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55c852a44fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x55c852a45d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x55c852b2a275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x55c852a058fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x55c852acf30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x55c852acf772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x55c8525c05a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x55c8525b7708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x55c85253c24e]
13: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0xc6) [0x55c852540646]
14: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x55c852530e5c]
15: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x55c852163d78]
16: (OSD::load_pgs()+0x4a0) [0x55c8520d9f80]
17: (OSD::init()+0x2217) [0x55c8520f5187]
18: (main()+0x2eb8) [0x55c852005538]
19: (__libc_start_main()+0xf1) [0x7f8a795ba3f1]
20: (_start()+0x29) [0x55c852090e69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 12:28:17.575591 7f8a7c391e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f8a7c391e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x55c8526752f4]
2: (()+0x11670) [0x7f8a7a63a670]
3: (()+0x1fd9f) [0x7f8a7bf6dd9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x55c852a44fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x55c852a45d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x55c852b2a275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x55c852a058fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x55c852acf30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x55c852acf772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x55c8525c05a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x55c8525b7708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x55c85253c24e]
13: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0xc6) [0x55c852540646]
14: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x55c852530e5c]
15: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x55c852163d78]
16: (OSD::load_pgs()+0x4a0) [0x55c8520d9f80]
17: (OSD::init()+0x2217) [0x55c8520f5187]
18: (main()+0x2eb8) [0x55c852005538]
19: (__libc_start_main()+0xf1) [0x7f8a795ba3f1]
20: (_start()+0x29) [0x55c852090e69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 13:42:39.014748 7f834d6b0e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f834d6b0e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x56472da232f4]
2: (()+0x11670) [0x7f834b959670]
3: (()+0x1fd9f) [0x7f834d28cd9f]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0x1d9) [0x56472ddefd39]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x3b7) [0x56472ddf1717]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x56472ddf1a5c]
7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x56472ddfa1d7]
8: (()+0xe5849e) [0x56472de2549e]
9: (()+0xe58566) [0x56472de25566]
10: (()+0xe586e1) [0x56472de256e1]
11: (rocksdb::MergingIterator::Next()+0x449) [0x56472de08839]
12: (rocksdb::DBIter::Next()+0xd3) [0x56472dea68a3]
13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x56472d963e7a]
14: (BlueStore::OmapIteratorImpl::next(bool)+0x38) [0x56472d8a5b18]
15: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+0xc65) [0x56472d588295]
16: (PG::read_state(ObjectStore
, ceph::buffer::list&)+0x399) [0x56472d535819]
17: (OSD::load_pgs()+0x8cd) [0x56472d4883ad]
18: (OSD::init()+0x2217) [0x56472d4a3187]
19: (main()+0x2eb8) [0x56472d3b3538]
20: (__libc_start_main()+0xf1) [0x7f834a8d93f1]
21: (_start()+0x29) [0x56472d43ee69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 13:42:39.014748 7f834d6b0e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f834d6b0e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x56472da232f4]
2: (()+0x11670) [0x7f834b959670]
3: (()+0x1fd9f) [0x7f834d28cd9f]
4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0x1d9) [0x56472ddefd39]
5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x3b7) [0x56472ddf1717]
6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x2ac) [0x56472ddf1a5c]
7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x97) [0x56472ddfa1d7]
8: (()+0xe5849e) [0x56472de2549e]
9: (()+0xe58566) [0x56472de25566]
10: (()+0xe586e1) [0x56472de256e1]
11: (rocksdb::MergingIterator::Next()+0x449) [0x56472de08839]
12: (rocksdb::DBIter::Next()+0xd3) [0x56472dea68a3]
13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x9a) [0x56472d963e7a]
14: (BlueStore::OmapIteratorImpl::next(bool)+0x38) [0x56472d8a5b18]
15: (void PGLog::read_log_and_missing<pg_missing_set<true> >(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&, pg_missing_set<true>&, bool, std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, bool, bool*, DoutPrefixProvider const*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)+0xc65) [0x56472d588295]
16: (PG::read_state(ObjectStore
, ceph::buffer::list&)+0x399) [0x56472d535819]
17: (OSD::load_pgs()+0x8cd) [0x56472d4883ad]
18: (OSD::init()+0x2217) [0x56472d4a3187]
19: (main()+0x2eb8) [0x56472d3b3538]
20: (__libc_start_main()+0xf1) [0x7f834a8d93f1]
21: (_start()+0x29) [0x56472d43ee69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 13:46:01.092574 7f0512573e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f0512573e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x562837da82f4]
2: (()+0x11670) [0x7f051081c670]
3: (()+0x1fd9f) [0x7f051214fd9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x562838177fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x562838178d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x56283825d275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x5628381388fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x56283820230e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x562838202772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x562837cf35a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x562837cea708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x562837c6f24e]
13: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0xc6) [0x562837c73646]
14: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x562837c63e5c]
15: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x562837896d78]
16: (OSD::load_pgs()+0x4a0) [0x56283780cf80]
17: (OSD::init()+0x2217) [0x562837828187]
18: (main()+0x2eb8) [0x562837738538]
19: (__libc_start_main()+0xf1) [0x7f050f79c3f1]
20: (_start()+0x29) [0x5628377c3e69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 13:46:01.092574 7f0512573e40 -1 ** Caught signal (Segmentation fault) *
in thread 7f0512573e40 thread_name:ceph-osd

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x562837da82f4]
2: (()+0x11670) [0x7f051081c670]
3: (()+0x1fd9f) [0x7f051214fd9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x562838177fbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x562838178d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x56283825d275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x5628381388fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x56283820230e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x562838202772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x562837cf35a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x562837cea708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x562837c6f24e]
13: (BlueStore::omap_get_values(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >)+0xc6) [0x562837c73646]
14: (BlueStore::omap_get_values(coll_t const&, ghobject_t const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::list> > >
)+0x5c) [0x562837c63e5c]
15: (PG::_has_removal_flag(ObjectStore*, spg_t)+0x148) [0x562837896d78]
16: (OSD::load_pgs()+0x4a0) [0x56283780cf80]
17: (OSD::init()+0x2217) [0x562837828187]
18: (main()+0x2eb8) [0x562837738538]
19: (__libc_start_main()+0xf1) [0x7f050f79c3f1]
20: (_start()+0x29) [0x5628377c3e69]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2017-09-07 13:57:25.827217 7f47d861e700 -1 ** Caught signal (Segmentation fault) *
in thread 7f47d861e700 thread_name:tp_osd_tp

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x56480b0402f4]
2: (()+0x11670) [0x7f47fb0b5670]
3: (()+0x1fd9f) [0x7f47fc9e8d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x56480b40ffbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x56480b410d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x56480b4f5275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x56480b3d08fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x56480b49a30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x56480b49a772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x56480af8b5a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x56480af82708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x56480af0724e]
13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x918) [0x56480af3d3d8]
14: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x56480af3f77e]
15: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x17c) [0x56480aaef31c]
16: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x58) [0x56480ac78cb8]
17: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x391) [0x56480ad87ea1]
18: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2c1) [0x56480ad96d21]
19: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x56480aca7060]
20: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x53d) [0x56480ac0bd9d]
21: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x56480aa906b9]
22: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x56480ad27f57]
23: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x56480aab7e3e]
24: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x56480b087e64]
25: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56480b08aea0]
26: (()+0x76da) [0x7f47fb0ab6da]
27: (clone()+0x5f) [0x7f47fa11dd7f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2017-09-07 13:57:25.827217 7f47d861e700 -1 ** Caught signal (Segmentation fault) *
in thread 7f47d861e700 thread_name:tp_osd_tp

ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)
1: (()+0xa562f4) [0x56480b0402f4]
2: (()+0x11670) [0x7f47fb0b5670]
3: (()+0x1fd9f) [0x7f47fc9e8d9f]
4: (rocksdb::BlockBasedTable::NewIndexIterator(rocksdb::ReadOptions const&, rocksdb::BlockIter*, rocksdb::BlockBasedTable::CachableEntry<rocksdb::BlockBasedTable::IndexReader>*)+0x5cc) [0x56480b40ffbc]
5: (rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, bool)+0x2a8) [0x56480b410d18]
6: (rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileDescriptor const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::HistogramImpl*, bool, int)+0x655) [0x56480b4f5275]
7: (rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, rocksdb::RangeDelAggregator*, bool*, bool*, unsigned long*)+0x43b) [0x56480b3d08fb]
8: (rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*)+0x57e) [0x56480b49a30e]
9: (rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*)+0x22) [0x56480b49a772]
10: (rocksdb::DB::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0xc1) [0x56480af8b5a1]
11: (RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const
, unsigned long, ceph::buffer::list*)+0x158) [0x56480af82708]
12: (BlueStore::Collection::get_onode(ghobject_t const&, bool)+0x5ae) [0x56480af0724e]
13: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x918) [0x56480af3d3d8]
14: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x56480af3f77e]
15: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x17c) [0x56480aaef31c]
16: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x58) [0x56480ac78cb8]
17: (ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x391) [0x56480ad87ea1]
18: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2c1) [0x56480ad96d21]
19: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x56480aca7060]
20: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x53d) [0x56480ac0bd9d]
21: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x56480aa906b9]
22: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x56480ad27f57]
23: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x56480aab7e3e]
24: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x56480b087e64]
25: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56480b08aea0]
26: (()+0x76da) [0x7f47fb0ab6da]
27: (clone()+0x5f) [0x7f47fa11dd7f]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues

Related to bluestore - Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc Closed 07/10/2017

History

#1 Updated by Sage Weil almost 2 years ago

  • Status changed from New to Duplicate

Pretty sure this is a dup of http://tracker.ceph.com/issues/21318 and http://tracker.ceph.com/issues/20557

Disable jemalloc in /etc/{sysconfig,default}/ceph and the problem should go away

If that's nto the problem, please let me know!

#2 Updated by Sage Weil almost 2 years ago

  • Related to Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc added

#3 Updated by Barry Stokes almost 2 years ago

jemalloc was not enabled when this was happening.

Also available in: Atom PDF