Ceph » Messengers

Right now, it seems a throttled connection will still receive a TCP receive buffer's worth of data, but because the userspace daemon isn't reading the socket, that data is kept in kernel memory. There's a global limit how much TCP memory can use for these buffers. Enough throttled connections with enough buffered will cause the available memory for other sockets to be low, and thus slowing down performance of the non-throttled connections.

Jim Schutt's email: http://marc.info/?l=ceph-devel&m=133009796706284&w=2

Ceph's throttling could use SO_RCVBUF (or some other such mechanism) to try to limit the amount of data buffered for throttled connections. A quick googling did not give me a good example of an established project using this technique, so we should tread carefully, and see what others are doing.