Ceph

At the moment, we do not track objects which are unstable due to a queued primary write. A client might then write (to the primary), get a commit, read from the replica, and get the old version because, while the replica has journaled the write, it has not yet applied it. The primary handles this with the ObjectContext rwstate tracker by blocking the read until the write is applied locally. The replica needs a similar machanism. Probably, we simply distinguish part of the ObjectContext as vaild/required on the replica and use the existing object context machinery (with no cache?).

A crucial part of this task is updating the ceph_test_rados tool to accept a read_from_replica option to enable the librados read from replica feature on some reads, as well as poking the functionality into the ceph-qa-suite rados.py task and some suite yamls.