Ceph

h1. Libradosobjecter - smarter localized reads

h3. Summary

Allow reads from object replicas on the same host, same rack, same data center, or other distance metric.

h3. Owners

* Name (Affiliation)

h3. Interested Parties

* Sage Weil (Inktank)

* Loic Dachary <loic@dachary.org>

* Sam Just (Inktank)

h3. Current Status

There is currently a librados/libcephfs/Objecter flag that tells the client to read from a non-primary replica if it happens to be on the same host (as indicated by a matching IP address).  This works in some situations but is otherwise pretty limited.  

h3. Detailed Description

Clusters typically have some sort of hierarchical structure, and we conveniently have that handy in the CRUSH map.  We want to use that information to choose a replica that is closes to us.

The key missing piece of information is where the client is in a form that can be reconciled with the CRUSH map.  If the client has a set of key/value pairs indicating its location in the same terms that and OSDs location is reflected by CRUSH, we can use that to choose the one that is closest to us. 

The 'crush location' config option would be a simple list of key/value pairs, e.g.

p((. crush location = host=foo rack=bar room=baz

We might allow multiple locations to be listed:

p((. crush location = host=foo host=foo2 rack=bar

which would be useful when there are parallel hierarchies and we want to indicate locality for both.

Objecter would look for the match with the CRUSH hierarchy with the lowest-valued crush type (i.e., a matching host is closer than a matching rack)

This would be triggered by the existing LOCALIZED_READS flag implemented in the Objecter and exposed to varying degrees by librados and libcephfs. 

h3. Work items

h3. Coding tasks

# Objecter: add 'crush location' config option and parse on init

# Objecter: add additonal API calls to adjust this location setting at runtime

# Objecter: choose the closest replica based on location information, when it is specified.  This can either supplement or (more likely) replace the current check for a matching IP address.

# librados, libcephfs: expose explicit API to set the location.  This would supplement simply setting the 'crush location = ...' config option

# [maybe] Update hadoop bindings to use the new API

# [maybe] librbd: set localized reads flag on clone parents?

h3. Build / release tasks

# Build/expand localized-reads test to verify the correct replica is chosen

h3. Documentation tasks

# Document API changes

h3. Tracker Links

http://tracker.ceph.com/issues/5035