Ceph » crimson

Also see https://pulpito.ceph.com/yingxin-2023-11-27_02:14:08-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467425/

https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467535/

https://pulpito.ceph.com/matan-2023-11-30_16:06:51-crimson-rados-wip-matanb-crimson-osdmap-trimming-distro-crimson-smithi/7472613

https://pulpito.ceph.com/matan-2023-11-30_16:06:51-crimson-rados-wip-matanb-crimson-osdmap-trimming-distro-crimson-smithi/7472643

https://pulpito.ceph.com/matan-2024-02-21_12:07:57-crimson-rados-wip-matanb-crimson-alien-buf-v3-testing-distro-crimson-smithi/7569495

OSDs 0,2 and 3

https://pulpito.ceph.com/sjust-2024-02-21_05:52:17-crimson-rados-wip-sjust-crimson-testing-2024-02-20-distro-default-smithi/7569188/

https://pulpito.ceph.com/sjust-2024-02-21_05:52:17-crimson-rados-wip-sjust-crimson-testing-2024-02-20-distro-default-smithi/7569205/

WaitSubop, WaitTrimTimmer, and WaitRepop are pipeline stages local to the
operation. As such they don't actually provide any ordering guarrantees as
only one operation will ever enter them. Rather, the intent is to hook into
the event system to expose information to an administrator.

This poses a problem for OrderedConcurrentPhase as it is currently implemented.
PipelineHandle::exit() is invoked prior to the op being destructed.
PipelineHandle::exit() does:

    void exit() {
        barrier.reset();
    }

For OrderedConcurrentPhase, ~ExitBarrier() invokes ExitBarrier::exit():

    void exit() final {
      if (barrier) {
        assert(phase);
        assert(phase->core == seastar::this_shard_id());
        std::ignore = std::move(*barrier
        ).then([phase=this->phase] {
          phase->mutex.unlock();
        });
        barrier = std::nullopt;
        phase = nullptr;
      } else if (phase) {
        assert(phase->core == seastar::this_shard_id());
        phase->mutex.unlock();
        phase = nullptr;
      }
    }

The problem comes in not waiting for the phase->mutex.unlock() to occur. For
SnapTrimEvent, phase is actually in the operation itself. It's possible for
that continuation

        ).then([phase=this->phase] {
          phase->mutex.unlock();

to occur after the last finally() in ShardServices::start_operation completes
and releases the final reference to SnapTrimEvent. This is harmless normally
provided that the PG or connection outlives it, but it's a problem for these
stages.

For now, let's just remove these stages. We can reintroduce another mechanism
later to set these event flags without an actual pipeline stage.

This is likely a bug even with pipelines not embedded in an operation, but we
can fix it later -- https://tracker.ceph.com/issues/64545.

Removing the above noop stages created a different problem.

SnapTrimEvent doesn't actually do or block on WaitForActive, RecoverMissing,
GetOBC, or Process. Entering Process, in
particular, causes problems unless we immediately leave it as
SnapTrimObjSubEvent needs to enter and leave it to complete. Entering one of
noop stages alluded to above had a side effect of exiting Process --
without that exit SnapTrimEvent and SnapTrimObjSubEvent mutually block
preventing snap trim or client io from making progress.

SnapTrimObjSubEvent doesn't need WaitForActive, PGActivationBlocker,
or RecoverMissing.

I'm working on a minor refactor to handle the above issues.

https://pulpito.ceph.com/yingxin-2024-04-02_02:43:05-crimson-rados-ci-yingxin-crimson-load-obc-3-base-distro-default-smithi/7635302/

https://pulpito.ceph.com/matan-2024-04-30_07:11:13-crimson-rados-wip-matanb-crimson-testing-user-modify-distro-crimson-smithi/7680800