Ceph

Shouldn't btrfs be able to detect that the disk is gone and return appropriate error codes itself, rather than hanging?

I have a Quarch box in the lab that I was just pointed to. It has an ssh interface to power cycle drives for failure testing (little shims that sit between the drive's sata/power and the system chassis, so I could automate some additional testing if we come up with a solution.
seems difficult to solve well without mandating an underlying filesystem. If btrfs is required for drive failure tolerance, then I imagine we can fix the btrfs handling of drive pull.

The only other way I can think of, is to have osds be informed of what actual devices they are using underneath, and then watching the sysfs entries for state changes on those drives. This would not detect/deal with filesystem level problems that aren't the result of a disk problem. For an integrated system/appliance, the sysfs monitoring would probably work. But for general use, something else needs to happen.

To keep the tradition of allowing any underlying filesystem, a load monitor/timeout should work. A large timeout would allow the system to become slow due to load without triggering an OSD exit. But it seems like a better solution would be to monitor activity and crash the osd only if there's no I/O completions and a request times out. I didn't notice the cosd's going into uninterruptible sleep (they were killable), so it'll probably work fairly cleanly.

We need to be ready to handle unresponsive FileStores in general. Even if the underlying filesystem is 100% perfect (ha ha), the hardware itself may have problems which cause unresponsiveness. So we need to handle it in our code.

merged by 6dc8994b750631c15e88553fd4fabdd9e4907989