Ceph » Messengers

According to http://pubs.opengroup.org/onlinepubs/009695399/functions/recv.html,

[EAGAIN] or [EWOULDBLOCK]
The socket's file descriptor is marked O_NONBLOCK and no data is waiting to be received; or
MSG_OOB is set and no out-of-band data is available and either the socket's file descriptor
is marked O_NONBLOCK or the socket does not support blocking to await out-of-band data.

That makes it a little more explicit than the Linux manpage does that we shouldn't get EAGAIN if we have pre-checked for data, but I'm nervous about the edges. Sage suggests printing out a warning if we get EAGAIN here; we could also make it some kind of config that defaults to looping on EAGAIN but lets us disable that if we find errors in the future?

Revert PR at https://github.com/ceph/ceph/pull/12374 may get more discussion of options.

Greg Farnum wrote:

So, I think this patch was a mistake and we need to revert it wherever it's appeared.

From "man 2 recv":
[...]

So, the only way we get EAGAIN is if we have no data available when making the ::recv() call, or it would block for another undetermined reason. But we always make sure data is available before doing invoking this function (by calling tcp_read_wait())! So marking a failure on EAGAIN is either blatantly incorrect, or masking some other kind of failure as turned out to be the root cause here (see #18184 for more details).

As I stated in commit message, tcp_read_wait() does not guarantee the socket has data (it return 0 if there are pending data). For example, the loop happens if Pipe::tcp_read() want to read 100 bytes, there are only 50 bytes pending data and the socket has no data.

Greg Farnum wrote:

That makes it a little more explicit than the Linux manpage does that we shouldn't get EAGAIN if we have pre-checked for data

It's perfectly fine to get EAGAIN in non-blocking mode (even if the socket is reportedly ready for read):

Under Linux, select() may report a socket file descriptor as "ready for reading",
while nevertheless a subsequent read blocks. This could for example happen when data
has arrived but upon examination has wrong checksum and is discarded. There may be other
circumstances in which a file descriptor is spuriously reported as ready.
Thus it may be safer to use O_NONBLOCK on sockets that should not block

https://linux.die.net/man/2/select