Ceph

Currently we use the ondisk_write/read locks to do mutual exclusion over the local filestore which avoids reading data that is being modified. We also need to block reads while we are waiting for replcias to commit. Otherwise object state can appear to go back in time.

- start a write locally, and on remotes
- finish local write
- another op reads (new) locally written value
- all osds crash
- replica osds restart, but old primary does not
- they recover
- client reads prior object value

we do not see this in our testing because on failure/peering our clients resubmit their requests.