Project

General

Profile

Actions
Copy link

Bug #43131

closed

segfault in BlueStore::Collection::split_cache()

Added by Kefu Chai over 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
32665
Crash signature (v1):
Crash signature (v2):

Description

2019-12-02T19:46:46.756+0000 7fa49ae03700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fa49ae03700 thread_name:tp_osd_tp

 ceph version 15.0.0-7893-g2b0f26e (2b0f26e1dc12ac42b2dc82e36f952be6b7dda10d) octopus (dev)
 1: (()+0x12890) [0x7fa4bad51890]
 2: (BlueStore::Collection::split_cache(BlueStore::Collection*)+0xb86) [0x55fdda805bb6]
 3: (BlueStore::_split_collection(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Collection>&, unsigned int, int)+0x161) [0x55fdda806b81]
 4: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x830) [0x55fdda830ce0]
 5: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2ae) [0x55fdda8338ae]
 6: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x80) [0x55fdda381c70]
 7: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x1e3) [0x55fdda3242a3]
 8: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1e1) [0x55fdda353431]
 9: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x50) [0x55fdda5829d0]
 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x55fdda345aac]
 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x55fdda983c1c]
 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55fdda986e70]
 13: (()+0x76db) [0x7fa4bad466db]
 14: (clone()+0x3f) [0x7fa4b9ae688f]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

/a/kchai-2019-12-02_17:51:44-rados-wip-kefu-testing-2019-12-02-2333-distro-basic-mira/4562085

/a/kchai-2019-12-02_17:51:44-rados-wip-kefu-testing-2019-12-02-2333-distro-basic-mira/4562251

Related issues 2 (0 open — 2 closed)

Related to bluestore - Bug #43217: segv in BlueStore::OnodeSpace::map_anyDuplicate

Actions
Related to bluestore - Bug #43147: segv in LruOnodeCacheShard::_pinResolvedMark Nelson

Actions
Actions
Copy link
 #1

Updated by Kefu Chai over 4 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Kefu Chai over 4 years ago

might be a regression introduced by https://github.com/ceph/ceph/pull/29687. need to rerun the test without it.

Actions
Copy link
 #3

Updated by Igor Fedotov over 4 years ago

  • Project changed from RADOS to bluestore
Actions
Copy link
 #5

Updated by Sage Weil over 4 years ago

  • Related to Bug #43217: segv in BlueStore::OnodeSpace::map_any added
Actions
Copy link
 #6

Updated by Sage Weil over 4 years ago

  • Related to Bug #43147: segv in LruOnodeCacheShard::_pin added
Actions
Copy link
 #7

Updated by Kefu Chai over 4 years ago 

2020-01-09T03:54:44.666+0000 7f0c90ab9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9142-g4359dde/rpm/el8/BUILD/ceph-15.0.0-9142-g4359dde/src/os/bluestore/BlueStore.cc: In function 'virtual void LruOnodeCacheShard::_add(BlueStore::OnodeRef&, int)' thread 7f0c90ab9700 time 2020-01-09T03:54:44.657678+0000
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9142-g4359dde/rpm/el8/BUILD/ceph-15.0.0-9142-g4359dde/src/os/bluestore/BlueStore.cc: 860: FAILED ceph_assert(o->s == nullptr)

 ceph version 15.0.0-9142-g4359dde (4359dde84d38890249873efbc290902c51796ba2) octopus (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x5599ed00d478]
 2: (()+0x507692) [0x5599ed00d692]
 3: (std::_Sp_counted_ptr_inplace<PriorityCache::Manager, std::allocator<PriorityCache::Manager>, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0) [0x5599ed5fd190]
 4: (BlueStore::Collection::split_cache(BlueStore::Collection*)+0x530) [0x5599ed5be060]
 5: (BlueStore::_split_collection(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Collection>&, unsigned int, int)+0x336) [0x5599ed5bf016]
 6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x9d8) [0x5599ed5dc838]
 7: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x3d8) [0x5599ed5df508]
 8: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x84) [0x5599ed15a5f4]
 9: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x102) [0x5599ed0f2c52]
 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x304) [0x5599ed11f8d4]
 11: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x5599ed34fff6]
 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1303) [0x5599ed1133d3]
 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x5599ed73d9f4]
 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5599ed73fcb4]
 15: (()+0x82de) [0x7f0cb82c62de]
 16: (clone()+0x43) [0x7f0cb7070a63]
Actions
Copy link
 #8

Updated by Sage Weil over 4 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 32665
Actions
Copy link
 #9

Updated by Sage Weil about 4 years ago 

Thread 1 (Thread 0x7f70f56b4700 (LWP 23017)):
#0  0x00007f711a6cdc6f in raise () from /lib64/libpthread.so.0
#1  0x000055889a735ed3 in reraise_fatal (signum=11) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/global/signal_handler.cc:326
#2  handle_fatal_signal (signum=11) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/global/signal_handler.cc:326
#3  <signal handler called>
#4  boost::intrusive::list_node_traits<void*>::set_next (next=0x5588a69356e8, n=0x0) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/build/boost/include/boost/intrusive/detail/list_node.hpp:65
#5  boost::intrusive::circular_list_algorithms<boost::intrusive::list_node_traits<void*> >::link_before (this_node=0x5588a69356e8, nxt_node=0x5588a670ebe8) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/build/boost/include/boost/intrusive/circular_list_algorithms.hpp:182
#6  boost::intrusive::list_impl<boost::intrusive::mhtraits<BlueStore::Onode, boost::intrusive::list_member_hook<>, &BlueStore::Onode::pin_item>, unsigned long, true, void>::push_front(BlueStore::Onode&) (value=..., this=0x5588a6192670)
    at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/build/boost/include/boost/intrusive/list.hpp:290
#7  LruOnodeCacheShard::_pin (this=0x5588a6190000, o=...) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.cc:898
#8  0x000055889a646396 in BlueStore::OnodeCacheShard::pin (o=..., this=0x5588a6190000) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.h:1214
#9  BlueStore::Onode::get (this=0x5588a6935600) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.h:1125
#10 0x000055889a610e66 in intrusive_ptr_add_ref (o=<optimized out>) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.h:3316
#11 boost::intrusive_ptr<BlueStore::Onode>::intrusive_ptr (rhs=..., this=0x7f70f56b0718) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:93
#12 BlueStore::Collection::split_cache (this=0x5588a63a7a40, dest=0x5588a8da6b40) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.cc:3724
#13 0x000055889a6122e6 in BlueStore::_split_collection (this=0x5588a60a0000, txc=0x5588a9226000, c=..., d=..., bits=5, rem=<optimized out>) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.cc:14747
#14 0x000055889a617b21 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /usr/include/c++/8/bits/stl_vector.h:930
#15 0x000055889a623ad8 in BlueStore::queue_transactions (this=0x5588a60a0000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x7f70f56b13c0) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/os/bluestore/BlueStore.cc:12216
#16 0x000055889a19c575 in ObjectStore::queue_transaction (this=0x5588a60a0000, ch=..., t=..., op=..., handle=0x7f70f56b13c0) at /usr/include/c++/8/bits/atomic_base.h:295
#17 0x000055889a139133 in OSD::dispatch_context (this=0x5588a5574000, ctx=..., pg=0x5588a66fe800, curmap=std::shared_ptr<const OSDMap> (use count 24, weak count 1) = {...}, handle=0x7f70f56b13c0) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:63
#18 0x000055889a166f78 in OSD::dequeue_peering_evt (this=0x5588a5574000, sdata=<optimized out>, pg=0x5588a66fe800, evt=std::shared_ptr<PGPeeringEvent> (use count 2, weak count 0) = {...}, handle=...) at /usr/include/c++/8/bits/shared_ptr.h:129
#19 0x000055889a398626 in ceph::osd::scheduler::PGPeeringItem::run (this=<optimized out>, osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at /usr/include/c++/8/ext/atomicity.h:96
#20 0x000055889a159f4f in ceph::osd::scheduler::OpSchedulerItem::run (handle=..., pg=..., sdata=<optimized out>, osd=<optimized out>, this=0x7f70f56b13f0) at /usr/include/c++/8/bits/unique_ptr.h:342
#21 OSD::ShardedOpWQ::_process (this=<optimized out>, thread_index=<optimized out>, hb=<optimized out>) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/osd/OSD.cc:10646
#22 0x000055889a786ea4 in ShardedThreadPool::shardedthreadpool_worker (this=0x5588a5574a28, thread_index=11) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/common/WorkQueue.cc:311
#23 0x000055889a789b04 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-9451.gb623579.el8.x86_64/src/common/WorkQueue.h:706
#24 0x00007f711a6c32de in start_thread () from /lib64/libpthread.so.0
#25 0x00007f711946d4b3 in clone () from /lib64/libc.so.6
Actions
Copy link
 #10

Updated by Sage Weil about 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF