Feature #3064

librbd: A way to read from nearby replicas

Added by Anonymous almost 8 years ago. Updated about 7 years ago.

Target version:
% Done:


Affected Versions:
Pull request ID:


In the style of CEPH_OSD_FLAG_LOCALIZE_READS, there are cases when an rbd image is stored in a pool where some of the replicas are closer than the others. That flag only does exact match to current host, it has no other metric of locality.

It would nice if a librados/librbd user could either:

- configure the client's location in terms of crush hierarchy (e.g. dc=us-west)
- configure a list of IP subnets that are considered "close"

These could even go in ceph.conf / libvirt rbd disk entry. "client location = dc:us-west" would match the syntax I've been playing with for OSDs to know what crush location to set at bootup. (Colon because I wanted to avoid "client_location=dc=us-west" confusion.)

I personally prefer the first one, because it isolates the client from knowing internal deployment details of the cluster. I don't know if the crush information would be available

And then, a flag in the api could actually enable reading from replicas; as it loosens the integrity guarantees, it should not be the default, or set for all clients in ceph.conf.

librbd could set the above flag by default for images that are marked "protected" (in the CoW sense), as those are known not to change.

Associated revisions

Revision 13ae13a9 (diff)
Added by Josh Durgin about 7 years ago

librbd: add options to enable balanced or localized reads for snapshots

Since snapshots never change, it's safe to read from replicas for them.
A common use for this would be reading from a parent snapshot shared by
many clones.

Convert LibrbdWriteback and AioRead to use the ObjectOperation api
so we can set flags. Fortunately the external wrapper holds no data,
so its lifecycle doesn't need to be managed.

Include a simple workunit that sets the flags in various combinations
and looks for their presence in the logs from 'rbd export'.

Fixes: #3064
Signed-off-by: Josh Durgin <>


#1 Updated by Neil Levine over 7 years ago

  • Status changed from New to 12

#2 Updated by Neil Levine over 7 years ago

OnAPP offer this with their SAN product:

#3 Updated by Ian Colle over 7 years ago

  • Target version set to v0.63

#4 Updated by Josh Durgin over 7 years ago

  • translation missing: en.field_story_points set to 5.00

#5 Updated by Ian Colle over 7 years ago

  • Status changed from 12 to In Progress

#6 Updated by Josh Durgin about 7 years ago

  • Status changed from In Progress to Resolved
  • Assignee set to Josh Durgin

Flags to do this for reads of snapshots are added by commit:13ae13a9068afcd4eb4b3574c46875cad8c91ab6.

Making the idea of 'localized' smarter can be implemented later, in #5035.

Also available in: Atom PDF