Ceph

I believe this is also a bug I'm seeing on OSX.

This is outside my area of expertise, but I'm looking at http://linux.die.net/man/3/poll and not seeing why POLLOUT should cause the behavior you're describing. Does its meaning vary on BSD systems, or can you tell me more about why it's a problem?
(I don't doubt that removing it removes the visible bug, but I want to understand the situation to make sure we aren't introducing more subtle issues!)

In signal_handler.cc at line 213, poll(2) returns immediately with POLLOUT set in revents for each of the struct pollfd entries, telling the process that each of those pipes may be written to without blocking. At lines 220 and 225 we try to read(2) from each of those pipes, but it fails because none of them have any data available for reading. Then we loop and try poll(2) again. We didn't do anything to change the state of the pipes, so they are still available for writing, and so poll(2) returns immediately again.

I don't know why the code works on Linux. Perhaps it's due to a difference in the way pipe buffering works? If FreeBSD allows a process to write to a pipe before any process has opened that pipe for reading, but Linux requires the pipe first be opened for reading, then that would explain the discrepancy. But that's just a guess, and not a very good one.

I found that the daemons on OSX were spinning not because they were not blocking in poll, but because the unnamed semaphore used in CephContextServiceThread is borked on OSX. Namely, sem_init() always returns an error. To fix, I used the Mach semaphores. Not sure if this is a problem on FreeBSD, or not.

Recent Ceph checkouts don't even compile on FreeBSD. Only the wip-port branch can compile. So I don't think there's any need to backport the fix to older Ceph versions.