Ceph » bluestore

But I'm seeing also those:

    0> 2018-10-22 15:40:34.772349 7f730dbfe700 -1 *** Caught signal (Segmentation fault) **
in thread 7f730dbfe700 thread_name:tp_osd_disk
ceph version 12.2.8-10-ga49f886acf (a49f886acf37d7254fc404807e4d26ffe16d3096) luminous (stable)
1: (()+0xa3b144) [0x560c1153e144]
2: (()+0x110c0) [0x7f737cd510c0]
3: (BlueStore::OldExtent::create(boost::intrusive_ptr<BlueStore::Collection>, unsigned int, unsigned int, unsigned int, boost::intrusive_ptr<BlueStore::Blob>&)+0x9a) [0x560c113e374a]
4: (BlueStore::ExtentMap::punch_hole(boost::intrusive_ptr<BlueStore::Collection>&, unsigned long, unsigned long, boost::intrusive::list<BlueStore::OldExtent, boost::intrusive::member_hook<BlueStore::OldExtent, boost::intrusive::list_member_hook<void, void, void>, &BlueStore::OldExtent::old_extent_item>, void, void, void>*)+0x211) [0x560c1141d681]
5: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::allocator<BlueStore::SharedBlob*> >*)+0x2a0) [0x560c114247c0]
6: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc5) [0x560c11425055]
7: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7b) [0x560c11426b7b]
8: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1f55) [0x560c11434475]
9: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x546) [0x560c11435186]
10: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x14f) [0x560c10fc6def]
11: (remove_dir(CephContext*, ObjectStore*, SnapMapper*, OSDriver*, ObjectStore::Sequencer*, coll_t, std::shared_ptr<DeletingState>, bool*, ThreadPool::TPHandle&)+0x567) [0x560c10f505a7]
12: (OSD::RemoveWQ::_process(std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, ThreadPool::TPHandle&)+0x1cc) [0x560c10f5146c]
13: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> >, std::pair<boost::intrusive_ptr<PG>, std::shared_ptr<DeletingState> > >::_void_process(void*, ThreadPool::TPHandle&)+0x122) [0x560c10fbd562]
14: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb8) [0x560c1158d1f8]
15: (ThreadPool::WorkThread::entry()+0x10) [0x560c1158e390]
16: (()+0x7494) [0x7f737cd47494]
17: (clone()+0x3f) [0x7f737bdceacf]

in thread 7fe09aff8700 thread_name:bstore_kv_final
ceph version 12.2.8-10-ga49f886acf (a49f886acf37d7254fc404807e4d26ffe16d3096) luminous (stable)
1: (()+0xa3b144) [0x560ebb7c1144]
2: (()+0x110c0) [0x7fe0ad5b20c0]
3: (()+0xe159) [0x7fe0af04b159]
4: (()+0x298c6) [0x7fe0af0668c6]
5: (free()+0x359) [0x7fe0af0414e9]
6: (std::__detail::_Hashtable_alloc<mempool::pool_allocator<(mempool::pool_index_t)4, std::__detail::_Hash_node<std::pair<ghobject_t const, boost::intrusive_ptr<BlueStore::Onode> >, true> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<ghobject_t const, boost::intrusive_ptr<BlueStore::Onode> >, true>*)+0x6b) [0x560ebb6ce5db]
7: (std::_Hashtable<ghobject_t, std::pair<ghobject_t const, boost::intrusive_ptr<BlueStore::Onode> >, mempool::pool_allocator<(mempool::pool_index_t)4, std::pair<ghobject_t const, boost::intrusive_ptr<BlueStore::Onode> > >, std::__detail::_Select1st, std::equal_to<ghobject_t>, std::hash<ghobject_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear()+0x23) [0x560ebb6ce663]
8: (BlueStore::OnodeSpace::clear()+0x1b2) [0x560ebb667542]
9: (BlueStore::_reap_collections()+0x383) [0x560ebb667973]
10: (BlueStore::_kv_finalize_thread()+0xbc9) [0x560ebb6774a9]
11: (BlueStore::KVFinalizeThread::entry()+0xd) [0x560ebb6d18ad]
12: (()+0x7494) [0x7fe0ad5a8494]
13: (clone()+0x3f) [0x7fe0ac62facf]

All from different servers and different OSDs.

The second log is similar to
http://tracker.ceph.com/issues/36526

Yes so my question is if all of those are may be just a result of the race mentioned here: https://github.com/ceph/ceph/pull/24701

Stefan Priebe wrote:

Yes so my question is if all of those are may be just a result of the race mentioned here: https://github.com/ceph/ceph/pull/24701

It could be. We'll get those 2 recent fixes backported shortly. How reproducible is this? Can we do a test branch/build on top of 12.2.9 for you to test?

Not that good ;-) it always happen, when we trigger a heavy backfill or recovery. But i don't want to pull that many disks ;-) We only had this while replacing old ssds with new ones.