Ceph

RADOS supports the ability to send reads to replicas rather than the primary. The primary use for this feature is to allow a client to read from a replica closer than the primary.

The rados suite has support for this feature in several tests (see yamls with balance_reads: true and localize_reads: true).

The above tests correctness, but it doesn't actually validate that we're serving those reads from the replica. There are cases where the replica is supposed to bounce the read back to the client to be redirected to the client (by returning -EAGAIN, see Objecter::handle_osd_op_reply), but the goal of this ticket is to create a test that creates a situation where we can really expect every read to be served by the replica and validate that it really does.

The core reason for a replica to bounce read to the primary is that the replica can't serve a read on the most recently modified objects in the PG. This isn't really an uncommon situation for ceph_test_rados, indeed it goes out of it's way to submit reads immediately after writes to ensure that the update is immediately available.

Thus, we'd like to add a test like this:
1. create rbd image1
2. perform 1 minute of random writes to image1
3. create rbd image 2
4. perform 1 minute of random writes to image2
5. perform 1 minutes of random reads from image 1 with balance_reads set
6. check that l_osdc_replica_read_bounced on the client in 5 is 0

rbd bench should be a good fit for the above, rbd can be configured to use balanced_reads.

The perf counter mentioned above is added as part of https://github.com/ceph/ceph/pull/56677, which also improves replica_read reliability.