Ceph

I've observed frequent system deadlocks while writing lots of data to the kernel-mounted ceph filesystem, which do not occur if I disable the osd on that host while uploading the files.

I suspect it has to do with sync() filesystem calls, that end up partially-blocking the osd and/or the monitor that calls sync() until it completes, which in turn prevents the ceph filesystem from flushing all data to disk: deadlock.

I suspect the new syncfs() calls may be a solution to this, but other than separating mon/osd and clients on separate (virtual?) machines, are there any other ways around this? Or is there more to the problem than just sync() calls?

E.g., at times I can't kill the monitor that ceph.ko has bound to, but even after kill -9 it keeps the channel to other monitors open, so they don't regard it as dead. I think this happens when the monitor calls sync() but the filesystem was not active due to a recovering mds. Recovery will only progress if I restart one of the other monitors, so that it starts and completes an election, taking over from the zombine monitor. (It doesn't help keep things running smoothly that the kernel client often fails to reconnect in time, and then its reconnect is denied, which leads to long waits, oopses and need for reboots).

This all was last observed with kernel 2.6.38.8, ceph 0.31, all btrfs, using separate volumes or separate filesystems for mon and osd.