librbd: A way to read from nearby replicas
In the style of CEPH_OSD_FLAG_LOCALIZE_READS, there are cases when an rbd image is stored in a pool where some of the replicas are closer than the others. That flag only does exact match to current host, it has no other metric of locality.
It would nice if a librados/librbd user could either:
- configure the client's location in terms of crush hierarchy (e.g. dc=us-west)
- configure a list of IP subnets that are considered "close"
These could even go in ceph.conf / libvirt rbd disk entry. "client location = dc:us-west" would match the syntax I've been playing with for OSDs to know what crush location to set at bootup. (Colon because I wanted to avoid "client_location=dc=us-west" confusion.)
I personally prefer the first one, because it isolates the client from knowing internal deployment details of the cluster. I don't know if the crush information would be available
And then, a flag in the api could actually enable reading from replicas; as it loosens the integrity guarantees, it should not be the default, or set for all clients in ceph.conf.
librbd could set the above flag by default for images that are marked "protected" (in the CoW sense), as those are known not to change.
librbd: add options to enable balanced or localized reads for snapshots
Since snapshots never change, it's safe to read from replicas for them.
A common use for this would be reading from a parent snapshot shared by
Convert LibrbdWriteback and AioRead to use the ObjectOperation api
so we can set flags. Fortunately the external wrapper holds no data,
so its lifecycle doesn't need to be managed.
Include a simple workunit that sets the flags in various combinations
and looks for their presence in the logs from 'rbd export'.
#2 Updated by Neil Levine over 7 years ago
OnAPP offer this with their SAN product: