Messenger - priorities for Client/Connection-orient¶
As we know, Ceph is aimed to provide with unified storage(S3, block, file) to users. When multi applications share the same storage infrastructure, they also share storage performance both IOPS and bandwidth. One application – or “noisy neighbor” – can easily consume an unfair share of the resources. Leaving mere scraps for others. We may expect ceph can eliminate resource contention and variable application performance caused by Noisy Neighbors.
In practice, different client should have different priority for processing request. For example, object storage like swift,s3 should be low priority compared to block storage. So users may expect if two request(one from radosgw, another from librbd) both got to the same osd for the same time, a higher priority connection is expected to processed firstly. Especially in high-bandwidth case, if we always try to process high priority connection, high priority connection can get better performance under full load env. Then if two message are waiting for reply in osd side when network send queue is full, the high priority connection is expected to be sent firstly.
Second, we may expect add some qos-field into message's header or footer, low priority connection's message replied from osd under full load will carry with "lower my data transmit" info.
The expected result picture is at associated file(see below)
Now we only have message priority which will affect recovery/client/scrub message processing. Actually all client op are the same now, message priority can't play well for truly qos.
Now AsyncMessenger uses multi dedicated threads served to all connections. In order to implement priority connection, each thread will be responsible for part of connections and each thread always read/write high priority connection's buffer data and do dispatch firstly. But if a low priority in one thread and a high priority in another thread, it still can't complete priority things. So we may expect AsyncMessenger maintain shared qos state to indicate each async worker thread can aware of other threads's status.
Client like librbd can set client priority level for each image, radosgw can set priority for different bucket and the same for cephfs.