Ceph

We need tests of OSD failures that verify the cluster is able to recover. Eventually this will need to be fleshed out to include a variety of failure scenarios that try to get good coverage on the peering and recovery code. We can start out with some pretty simple tests, though:

- restart an osd once, or every few minutes.  verify get back to all active+clean.  maybe within some time bound?
 - stop an osd, mark it out.  continue operation for a while (dirty lots of objects).  then re-add the osd.  (this exercises code paths similar to a regular cluster expansion)
 - restart multiple (or all) osds simultaneously.