Ceph

Currently clients, daemons, monitor peers, and admins all use the same monitor IP to communicate.

Extend the monmap to include additional addresses for each mon:

client: the existing client-facing interface
 cluster: an interface for peer monitors
 service: an interface for use by osds, and mdss (internal cluster services)
 admin: an interface for cluster administrative commands (TBD: how do we define this subset? just auth stuff?)

This would allow internal cluster communications to be completely independent of the client-facing network (and client-facing network infrastructure). it may also help in isolating network traffic/load between clients and internal services (osds, mdss).

The 'service' vs 'client' distinction is questionable, since osds and mdss (and radosgw even) are still consumers of the mon's services. but in the context of an entire cluster, this is probably still useful. in the rgw case, it might be useful to configure whether it uses the 'service' vs 'client' address (it is a normal librados user, after all).

This feature request came from a user who wants to build a ceph appliance and wants to isolate the internal cluster services from the client network infrastructure.