Ceph

Maybe we could use "ceph osd set noscrub" or "ceph osd unset noscrub" to achieve this.

https://github.com/ceph/ceph/pull/4138 is pending for review.

We have set "ceph osd set noscrub" and "ceph osd set no-deep-scrub" flags:

noscrub,nodeep-scrub flag(s) set

And PGs are keep getting deep scrubbed. There is a rebalance (backfilling) going on. IIRC deep scrubs get started to verify integrity of the PG on the new OSD. Is that (still) true? I don't think this makes sense anymore with bluestore ... as it's already doing CRC checks etc. while doing the backfilling, right? If not, it would make sense to do this instead of doing this work twice.

Anyway ... when those flags are set the cluster should NOT keep scheduling deep-scrubs ... and it's still doing this ... really annoying because it slows down backfilling, potentially triggering SLOW_OPS (in our case it does) and or even cause the "OSD::osd_op_tp thread" to timeout ... leading to issue #41255 for at least Mimic (13.2.8). When backfilling is done the balancer might want to tune things ... but won't run because (deep-)scrubs ...

I filed a related issue that scrubbing apparently cannot be drained from a cluster:

https://tracker.ceph.com/issues/59512?next_issue_id=59511