Ceph

After reading the documentation, I'm still missing some knowledge on how min_size affects I/O to a HEALTH_OK cluster.

This thread:
https://www.spinics.net/lists/ceph-users/msg35128.html

Explains clearly that a write must complete on <min_size> failure domain before being ACK to the client, which means that in-flight I/O are safe.

To quote Kjetil Jørgensen:

size is how many replicas of an object to maintain, min_size is how many
writes need to succeed before the primary can ack the operation to the client.

larger min_size most likely higher latency for writes.

Here is a quote from the documentation at http://docs.ceph.com/docs/master/rados/operations/pools/ :

min_size: Sets the minimum number of replicas required for I/O. See Set the Number of Object Replicas for further details. Replicated pools only.
This ensures that no object in the data pool will receive I/O with fewer than min_size replicas.

This description does not guarantee that a client will receive ACK from <min_size> OSD if the failure occurs after the write started.

I'd suggest updating the documentation to what is written in the mailing list (or a corrected version if it is wrong).