Schedule scrubbing by considering PG's last_scrub_timestamp globally
Copy the email from @ceph-devel to open a tracker:
Another potential problem with scrub scheduling, as observed in our deployment (2PB cluster, 70% full), was that some PGs hadn't been scrubbed for 1.5 months, even we have the configuration to do deep scrubbing weekly.
With our deployment and percentage of full of the cluster, as well as the conservative setting for scrubbing (osd_max_scrubs = 1), one round of scrubbing would not finish in 1 one week, so that we properly should schedule that monthly (with weekly shallow scrubbing).
Another problem, is that currently the scheduling of scrub is optimized locally at each OSD, that is, for each PG this OSD acts as the primary, it selects the one which hasn't been scheduled to scrubbing longest, put it as the candidate and request scrub reserver from all replicas. Since each OSD can only have 1 active scrubbing, that active slot could potentially always occupied by a replica, as a result, the PG whose primary is this OSD, fail to schedule and left behind.
Is this issue worth an enhancement?
[sage] Good point. Yeah, I think it's definitely worth fixing!
#2 Updated by Guang Yang over 4 years ago
The easiest way I can think of is to add a timestamp field into the reservation request, and for the OSD handling the reservation request, it checks the timestamp against its own primary PGs, and reject the reservation request if has a later timestamp, so that the local tick can get a chance to schedule its own scrubbing?
Does that look correct?