Feature #40986: cephfs qos: implement cephfs qos base on tokenbucket algorighm - CephFS - Ceph

Actions

Copy link

Feature #40986

open

cephfs qos: implement cephfs qos base on tokenbucket algorighm

Added by songbo wang almost 5 years ago. Updated almost 2 years ago.

Status:

Fix Under Review

Priority:

Normal

Assignee:

songbo wang

Category:

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

Reviewed:

Affected Versions:

Component(FS):

Client, Common/Protocol, MDS

Labels (FS):

Pull request ID:

29266

Description

The basic idea is as follows:

Set QoS info as one of the dir's xattrs;
All clients that can access the same dirs with the same QoS setting.
Follow the Quota's config flow. when the MDS receives the QoS setting, it'll also broadcast the message to all clients.
We can change the limit online.
[support]:
limit && burst config

[usage]:
setfattr -n ceph.qos.limit.iops -v 200 /mnt/cephfs/testdirs/
setfattr -n ceph.qos.burst.read_bps -v 200 /mnt/cephfs/testdirs/
getfattr -n ceph.qos.limit.iops /mnt/cephfs/testdirs/
getfattr -n ceph.qos /mnt/cephfs/testdirs/

[problems]:
Because there is no queue in CephFS IO path, If the bps is lower than the request's block size, the whole Client will be blocked until it gets enough token.

Actions

Copy link

Updated by songbo wang almost 5 years ago

I think there are two kinds of design:
1. all clients use the same QoS setting, just as the implementation
in this PR. Maybe there are multiple mount points, if we limit the
total IO, the number of total mount points is also limited.
So in my implementation, the total IO & BPS is not limited.

2. all clients share a specific QoS setting. I think there are two
kinds of use cases in detail.
2.1 setting a total limit, all clients limited by the average:
total_limit/clients_num.
2.2 setting a total limit, the mds decide the client's limitation by
their historical IO&BPS.

Actions

Copy link