Ceph » bluestore

lots of threads deleting objects. once of them:

  -821> 2019-09-23T03:53:40.621+0000 7f94d39ec700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f94d39ec700 thread_name:tp_osd_tp

 ceph version 15.0.0-5362-gf2f01cb (f2f01cb62f1119c0a348b00469bbdd6ffd9e6412) octopus (dev)
 1: (()+0xf5d0) [0x7f94f6c6f5d0]
 2: (BlueStore::OnodeSpace::lookup(ghobject_t const&)+0x1eb) [0x5583fb2e0f4b]
 3: (BlueStore::Collection::get_onode(ghobject_t const&, bool, bool)+0x70) [0x5583fb2ed500]
 4: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x1d26) [0x5583fb340b76]
 5: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x370) [0x5583fb356300]
 6: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x81) [0x5583faf00071]
 7: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x100) [0x5583faea2a00]
 8: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b8) [0x5583faece0b8]
 9: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&)+0x21f) [0x5583faece47f]
 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x143a) [0x5583faec2d4a]
 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x5583fb48c1d6]
 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5583fb48ed20]
 13: (()+0x7dd5) [0x7f94f6c67dd5]
 14: (clone()+0x6d) [0x7f94f5b2e02d]

bt

(gdb) bt
#0  0x00007f94f6c6f49b in raise () from /lib64/libpthread.so.0
#1  0x00005583fb43ca42 in reraise_fatal (signum=11) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/global/signal_handler.cc:326
#2  handle_fatal_signal(int) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/global/signal_handler.cc:326
#3  <signal handler called>
#4  0x00005583fb2e0f4b in set_next (n=<synthetic pointer>, next=<synthetic pointer>) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:851
#5  link_before (this_node=<synthetic pointer>, nxt_node=<optimized out>) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/build/boost/include/boost/intrusive/circular_list_algorithms.hpp:182
#6  push_front (value=..., this=0x558406ef1450) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/build/boost/include/boost/intrusive/list.hpp:290
#7  _touch (o=..., this=0x558406ef0000) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:851
#8  BlueStore::OnodeSpace::lookup(ghobject_t const&) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:1627
#9  0x00005583fb2ed500 in BlueStore::Collection::get_onode(ghobject_t const&, bool, bool) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:3629
#10 0x00005583fb340b76 in BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:12329
#11 0x00005583fb356300 in BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ()
    at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/os/bluestore/BlueStore.cc:12114
#12 0x00005583faf00071 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) (this=0x558406212000, ch=..., t=<optimized out>, op=..., handle=0x7f94d39e9700)
    at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/common/TrackedOp.h:385
#13 0x00005583faea2a00 in OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/osd/OSD.cc:9289
#14 0x00005583faece0b8 in OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/osd/OSD.cc:9860
#15 0x00005583faece47f in OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/osd/PeeringState.h:515
#16 0x00005583faec2d4a in run (handle=..., pg=..., sdata=<optimized out>, osd=<optimized out>, this=0x7f94d39e9730) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/osd/OpQueueItem.h:134
#17 OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/osd/OSD.cc:11021
#18 0x00005583fb48c1d6 in ShardedThreadPool::shardedthreadpool_worker(unsigned int) () at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/common/WorkQueue.cc:311
#19 0x00005583fb48ed20 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-5362-gf2f01cb/src/common/WorkQueue.h:706

several other threads also doing delete_work.

/a/sage-2019-09-23_02:45:54-rados-wip-sage2-testing-2019-09-22-1659-distro-basic-smithi/4327892 has the core and log.