Ceph

This should be really easy to implement using the Throttle class in the io submission methods (need to do both sync and async).

Maybe request count and byte count? g_conf.objecter_max_requests, object_max_request_bytes ?

I'm not sure this is actually feasible via the current objecter interface. It doesn't do much locking on its own and the layer above it (ie, the Client) does a lot of locking around Objecter calls so the Objecter can't really block. Without being able to block it can't slow down the rate of data input -- it can slow down how quickly it sends data out over the wire, but then it's just building up memory pressure locally. :/

After talking with Sage we're going to implement accounting in a non-blocking fashion in the Objecter, and then add a throttle function which Objecter's users can call prior to a read/write if they wish to and can assure that their locking will work out okay.
This throttle function will unlock Objecter::client_lock if it blocks on the throttle call and re-lock it before returning.

merged in ba1af748dc27285c561eb947e75e396229c342b8