Feature #11017
openImprove scrubbing throutput
0%
Description
Every OSD has a configurable scrubbing slot, ideally the slot should be taken with an active scrubbing if there are PGs in the queue for being scrubbing. In our cluster, we found that around a half of the slots are idle even there are PGs in the queue for scrubbing. More details:
We have 540 OSDs, the data pool is EC with 8 + 3 = 11 replicas. Ideally we should have 540/11 ~ 50 active scrubbings, however, we only have 20 at a maximum with sometime monitoring, even there are PGs in the queue.
Configuration:
"osd_max_scrubs": "1"
-bash-4.1$ sudo ceph -s cluster 035b9c00-3fd0-4123-a92f-778ce59a426e health HEALTH_OK monmap e2: 3 mons at {mon01c003=10.214.146.208:6789/0,mon02c003=10.214.147.130:6789/0,mon03c003=10.214.147.80:6789/0}, election epoch 48, quorum 0,1,2 mon01c003,mon03c003,mon02c003 osdmap e5568: 540 osds: 540 up, 540 in pgmap v10510804: 11424 pgs, 9 pools, 1429 TB data, 853 Mobjects 2057 TB used, 883 TB / 2941 TB avail 20 active+clean+scrubbing+deep 11404 active+clean
I think it makes sense to do the scheduling to potentially maximum the throughput (in terms of the number of active scrubbings).
Updated by Xinze Chi about 9 years ago
Are that some of the pgs interval of scrub small than osd_scrub_min_interval, so it reject it?
Updated by Guang Yang about 9 years ago
Xinze Chi wrote:
Are that some of the pgs interval of scrub small than osd_scrub_min_interval, so it reject it?
No. I think it is because there is not a PG that its OSDs have slots, for example, there is no scrub at osd.X, however, when it tries to kick off scrubbing, all its peers's slots have been taken..
Updated by Xinze Chi about 9 years ago
It is strange.
For example in your cluster, 20 active scrubbings? so there are 20 * 11 = 220 osds doing scrub at the same time. So there
are also (540 - 220) = 320 osd which are not doing scrub. I think there should be some others pgs only in these 320 osds
which could start a new scrub if load and time is permitted.
Updated by Guang Yang about 9 years ago
My math is something like:
When selecting the first PG for scrubbing, it has the chance of 540/450, second has the chance of (540 - 11)/540, and so on...
That gives 540/540 + (540 - 11)/540 + (540 - 11*2)/540 + ... = 24, pretty close to our observation..
Updated by Guang Yang about 9 years ago
- Status changed from New to Rejected
Thus I am thinking there is actually not much room for improvement ...
Updated by Guang Yang about 9 years ago
- Status changed from Rejected to New
Discussed with Xinze offline, he has some idea to improve, re-open for tracking.
Updated by Xinze Chi about 9 years ago
Now PR https://github.com/ceph/ceph/pull/3984 is pending for review.