Project

General

Profile

Actions
Copy link

Feature #64086

closed

Enable multicore messanger

Added by Matan Breizman 4 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Matan Breizman
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:
55641

Description

Set SocketMessenger::dispatch_only_on_sid to be false on "cluster" and "client" messengers.

See (Socket.cc):

seastar::future<ShardedServerSocket*>
ShardedServerSocket::create(bool dispatch_only_on_this_shard)
{
  auto primary_sid = seastar::this_shard_id();
  // start the sharded service: we should only construct/stop shards on #0
  return seastar::smp::submit_to(0, [primary_sid, dispatch_only_on_this_shard] {
    auto service = std::make_unique<sharded_service_t>();
    return service->start(
        primary_sid, dispatch_only_on_this_shard, construct_tag{}                  <-----
    ).then([service = std::move(service)]() mutable {
      auto p_shard = service.get();
      p_shard->local().service = std::move(service);
      return p_shard;
    });
  }).then([](auto p_shard) {
    return &p_shard->local();
  });
}

  using sharded_service_t = seastar::sharded<ShardedServerSocket>;
  std::unique_ptr<sharded_service_t> service;

This will set ShardedServerSocket::dispatch_only_on_primary_sid to be true when starting and creating the sharded service.
ShardedServerSocket::dispatch_only_on_primary_sid will be used in 2 cases

1) SocketMessenger::start:

  bool is_fixed_shard_dispatching() const {
    return dispatch_only_on_primary_sid;
  }


seastar::future<> SocketMessenger::start(
    const dispatchers_t& _dispatchers) {
  assert(seastar::this_shard_id() == sid);

  dispatchers.assign(_dispatchers);
  if (listener) {
    return listener->accept([this](SocketRef _socket, entity_addr_t peer_addr) {
      ..
      if (listener->is_fixed_shard_dispatching()) {                               <-----
        return accept(std::move(socket), peer_addr);
      } else {
        return seastar::smp::submit_to(sid,
            [this, peer_addr, socket = std::move(socket)]() mutable {
          return accept(std::move(socket), peer_addr);
        });
      }
    });
  }
  return seastar::now();
}

2) ShardedServerSocket::listen:

ShardedServerSocket::listen(entity_addr_t addr)
{
  ceph_assert_always(seastar::this_shard_id() == primary_sid);
  logger().debug("ShardedServerSocket({})::listen()...", addr);
    ..
    seastar::listen_options lo;
    if (ss.dispatch_only_on_primary_sid) {                      <-----
      lo.set_fixed_cpu(ss.primary_sid);
    }
    ss.listener = seastar::listen(s_addr, lo)

Actions
Copy link

Also available in: Atom PDF