Actions
Feature #64086
closedEnable multicore messanger
% Done:
0%
Source:
Tags:
Backport:
Description
Set SocketMessenger::dispatch_only_on_sid to be false on "cluster" and "client" messengers.
- Not supported yet:
- A crash can be reproduced with a 3 OSD cluster with smp 3 by running `ceph_test_cls_rbd`.
- https://pulpito.ceph.com/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/
See (Socket.cc):
seastar::future<ShardedServerSocket*>
ShardedServerSocket::create(bool dispatch_only_on_this_shard)
{
auto primary_sid = seastar::this_shard_id();
// start the sharded service: we should only construct/stop shards on #0
return seastar::smp::submit_to(0, [primary_sid, dispatch_only_on_this_shard] {
auto service = std::make_unique<sharded_service_t>();
return service->start(
primary_sid, dispatch_only_on_this_shard, construct_tag{} <-----
).then([service = std::move(service)]() mutable {
auto p_shard = service.get();
p_shard->local().service = std::move(service);
return p_shard;
});
}).then([](auto p_shard) {
return &p_shard->local();
});
}
using sharded_service_t = seastar::sharded<ShardedServerSocket>;
std::unique_ptr<sharded_service_t> service;
This will set ShardedServerSocket::dispatch_only_on_primary_sid to be true when starting and creating the sharded service.
ShardedServerSocket::dispatch_only_on_primary_sid will be used in 2 cases
1) SocketMessenger::start:
bool is_fixed_shard_dispatching() const {
return dispatch_only_on_primary_sid;
}
seastar::future<> SocketMessenger::start(
const dispatchers_t& _dispatchers) {
assert(seastar::this_shard_id() == sid);
dispatchers.assign(_dispatchers);
if (listener) {
return listener->accept([this](SocketRef _socket, entity_addr_t peer_addr) {
..
if (listener->is_fixed_shard_dispatching()) { <-----
return accept(std::move(socket), peer_addr);
} else {
return seastar::smp::submit_to(sid,
[this, peer_addr, socket = std::move(socket)]() mutable {
return accept(std::move(socket), peer_addr);
});
}
});
}
return seastar::now();
}
2) ShardedServerSocket::listen:
ShardedServerSocket::listen(entity_addr_t addr)
{
ceph_assert_always(seastar::this_shard_id() == primary_sid);
logger().debug("ShardedServerSocket({})::listen()...", addr);
..
seastar::listen_options lo;
if (ss.dispatch_only_on_primary_sid) { <-----
lo.set_fixed_cpu(ss.primary_sid);
}
ss.listener = seastar::listen(s_addr, lo)
Updated by Yingxin Cheng 3 months ago
When dispatch_only_on_primary_sid is set to true, the messenger is single-sharded -- it only dispatches connections in the same shard where the messenger is created.
When dispatch_only_on_primary_sid is set to false, the messenger becomes multiple-sharded -- it will dispatch connections (to the OSD) from all the available shards.
Updated by Matan Breizman 3 months ago
Full bt:
#0 0x00000000091ed6e7 in seastar::smp::submit_to<seastar::foreign_ptr<seastar::temporary_buffer<char> >::destroy_on(seastar::temporary_buffer<char>, unsigned int)::{lambda()#1}>(unsigned int, seastar::smp_submit_to_options, std::invoke_result&&) (func=..., options=..., t=2) at /***/ceph/src/seastar/include/seastar/core/smp.hh:309
#1 seastar::smp::submit_to<seastar::foreign_ptr<seastar::temporary_buffer<char> >::destroy_on(seastar::temporary_buffer<char>, unsigned int)::{lambda()#1}>(unsigned int, std::invoke_result&&) (func=..., t=2)
at /***/ceph/src/seastar/include/seastar/core/smp.hh:380
#2 seastar::foreign_ptr<seastar::temporary_buffer<char> >::destroy_on (cpu=2, p=...)
at /***/ceph/src/seastar/include/seastar/core/sharded.hh:859
#3 seastar::foreign_ptr<seastar::temporary_buffer<char> >::destroy (cpu=2, p=..., this=0x60c000174e18)
at /***/ceph/src/seastar/include/seastar/core/sharded.hh:848
#4 seastar::foreign_ptr<seastar::temporary_buffer<char> >::~foreign_ptr (this=0x60c000174e18,
__in_chrg=<optimized out>) at /***/ceph/src/seastar/include/seastar/core/sharded.hh:889
#5 ceph::buffer::raw_seastar_foreign_ptr::~raw_seastar_foreign_ptr (this=0x60c000174dc0,
__in_chrg=<optimized out>) at /***/ceph/src/common/buffer_seastar.cc:25
#6 0x00000000091ef39f in ceph::buffer::raw_seastar_foreign_ptr::~raw_seastar_foreign_ptr (this=0x60c000174dc0,
__in_chrg=<optimized out>) at /***/ceph/src/common/buffer_seastar.cc:25
#7 0x0000000006af8dd0 in ceph::buffer::v15_2_0::ptr::release (this=this@entry=0x603000309738)
at /***/ceph/src/common/buffer.cc:429
#8 0x000000000b483c17 in ceph::buffer::v15_2_0::ptr::~ptr (this=0x603000309738, __in_chrg=<optimized out>)
at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/list.tcc:66
#9 ceph::buffer::v15_2_0::ptr_node::~ptr_node (this=0x603000309730, __in_chrg=<optimized out>)
at /***/ceph/src/include/buffer.h:371
#10 ceph::buffer::v15_2_0::ptr_node::disposer::operator() (delete_this=0x603000309730, this=0x7f4454e6cce0)
--Type <RET> for more, q to quit, c to continue without paging--c
/ceph/src/include/buffer.h:366
#11 ceph::buffer::v15_2_0::list::buffers_t::clear_and_dispose (this=0x608000d1efe0) at /***/ceph/src/include/buffer.h:600
#12 ceph::buffer::v15_2_0::list::~list (this=0x608000d1efe0, __in_chrg=<optimized out>) at /***/ceph/src/include/buffer.h:959
#13 bluestore_deferred_op_t::~bluestore_deferred_op_t (this=0x608000d1efb0, __in_chrg=<optimized out>) at /***/ceph/src/os/bluestore/bluestore_types.h:1132
#14 std::destroy_at<bluestore_deferred_op_t> (__location=0x608000d1efb0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_construct.h:88
#15 std::allocator_traits<std::allocator<std::_List_node<bluestore_deferred_op_t> > >::destroy<bluestore_deferred_op_t> (__p=0x608000d1efb0, __a=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:537
#16 std::__cxx11::_List_base<bluestore_deferred_op_t, std::allocator<bluestore_deferred_op_t> >::_M_clear (this=this@entry=0x608000d1f228) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/list.tcc:77
#17 0x000000000b72ca5a in std::__cxx11::_List_base<bluestore_deferred_op_t, std::allocator<bluestore_deferred_op_t> >::~_List_base (this=0x608000d1f228, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_list.h:498
#18 std::__cxx11::list<bluestore_deferred_op_t, std::allocator<bluestore_deferred_op_t> >::~list (this=0x608000d1f228, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_list.h:827
#19 bluestore_deferred_transaction_t::~bluestore_deferred_transaction_t (this=0x608000d1f220, __in_chrg=<optimized out>) at /***/ceph/src/os/bluestore/bluestore_types.h:1155
#20 BlueStore::TransContext::~TransContext (this=0x618000188480, __in_chrg=<optimized out>) at /***/ceph/src/os/bluestore/BlueStore.h:1908
#21 0x000000000b18e26c in BlueStore::TransContext::~TransContext (this=0x618000188480, __in_chrg=<optimized out>) at /***/ceph/src/os/bluestore/BlueStore.h:1902
#22 BlueStore::_txc_finish (this=this@entry=0x621000042500, txc=<optimized out>) at /***/ceph/src/os/bluestore/BlueStore.cc:14169
#23 0x000000000b196129 in BlueStore::_txc_state_proc (this=this@entry=0x621000042500, txc=<optimized out>, txc@entry=0x618000188480) at /***/ceph/src/os/bluestore/BlueStore.cc:13853
#24 0x000000000b1a2aba in BlueStore::_kv_finalize_thread (this=this@entry=0x621000042500) at /***/ceph/src/os/bluestore/BlueStore.cc:14719
#25 0x000000000b7926c7 in BlueStore::KVFinalizeThread::entry (this=<optimized out>) at /***/ceph/src/os/bluestore/BlueStore.h:2240
#26 0x00000000116bda24 in Thread::entry_wrapper (this=this@entry=0x621000042dc0) at /***/ceph/src/common/Thread.cc:87
#27 0x00000000116bdbaf in Thread::_entry_func (arg=0x621000042dc0) at /***/ceph/src/common/Thread.cc:74
#28 0x00007f447cf301ca in start_thread () from /lib64/libpthread.so.0
Updated by Matan Breizman about 2 months ago
- Tracker changed from Support to Feature
- Status changed from In Progress to Resolved
- Assignee set to Matan Breizman
- Pull request ID changed from 52295 to 55641
Actions