Feature #64086
Updated by Matan Breizman 4 months ago
Set SocketMessenger::dispatch_only_on_sid to be false on "cluster" and "client" messengers.
* Not supported yet:
- A crash can be reproduced with a 3 OSD cluster with smp 3 by running `ceph_test_cls_rbd`.
- https://pulpito.ceph.com/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/
See (Socket.cc):
<pre><code class="text">
seastar::future<ShardedServerSocket*>
ShardedServerSocket::create(bool dispatch_only_on_this_shard)
{
auto primary_sid = seastar::this_shard_id();
// start the sharded service: we should only construct/stop shards on #0
return seastar::smp::submit_to(0, [primary_sid, dispatch_only_on_this_shard] {
auto service = std::make_unique<sharded_service_t>();
return service->start(
primary_sid, dispatch_only_on_this_shard, construct_tag{} <-----
).then([service = std::move(service)]() mutable {
auto p_shard = service.get();
p_shard->local().service = std::move(service);
return p_shard;
});
}).then([](auto p_shard) {
return &p_shard->local();
});
}
</code></pre>
<pre><code class="text">
using sharded_service_t = seastar::sharded<ShardedServerSocket>;
std::unique_ptr<sharded_service_t> service;
</code></pre>
This will set ShardedServerSocket::dispatch_only_on_primary_sid to be true when starting and creating the sharded service.
ShardedServerSocket::dispatch_only_on_primary_sid will be used in 2 cases
*1) SocketMessenger::start:
*<pre><code class="text">
bool is_fixed_shard_dispatching() const {
return dispatch_only_on_primary_sid;
}
</code></pre>
<pre><code class="text">
seastar::future<> SocketMessenger::start(
const dispatchers_t& _dispatchers) {
assert(seastar::this_shard_id() == sid);
dispatchers.assign(_dispatchers);
if (listener) {
return listener->accept([this](SocketRef _socket, entity_addr_t peer_addr) {
..
if (listener->is_fixed_shard_dispatching()) { <-----
return accept(std::move(socket), peer_addr);
} else {
return seastar::smp::submit_to(sid,
[this, peer_addr, socket = std::move(socket)]() mutable {
return accept(std::move(socket), peer_addr);
});
}
});
}
return seastar::now();
}
</code></pre>
*2) ShardedServerSocket::listen:*
<pre><code class="text">
ShardedServerSocket::listen(entity_addr_t addr)
{
ceph_assert_always(seastar::this_shard_id() == primary_sid);
logger().debug("ShardedServerSocket({})::listen()...", addr);
..
seastar::listen_options lo;
if (ss.dispatch_only_on_primary_sid) { <-----
lo.set_fixed_cpu(ss.primary_sid);
}
ss.listener = seastar::listen(s_addr, lo)
</code></pre>