Ceph » rbd

Currently, rbd_mirror_max_fetch_bytes parameter indicates maximum bytes to be read from each journal data object per fetch by the rbd-mirror daemon. Default value is 32k. So if there is need to fetch data from 'n' journal objects across the cluster then 32k*n bytes are fetched at one go by the rbd-mirror daemon. Given that rbd-mirroring is configured across two different ceph clusters across a WAN , one can expect a RTT of anywhere from 100 to 400ms on an average per fetch. Assuming, 1 Gb n/w across clusters and RTT of 400ms, Bandwidth product delay is around 50MB between two sites. For efficient use of network that customers pay to maintain connectivity between sites we need to transmit 50MB data in flight. If not, this may also result in missing meaningful SLAs due to prolonged slow replication.

One way to make effective use of network bandwidth is to increase this value from default 32k to higher value. In order to tune this parameter effectively, one needs to know the number of journal objects to be fetched in advance. Determining this number in advance is impossible in a dynamic cluster. For starters, one cannot determine the number of VMs carrying out I/O operations at any given point in time, and that's the number needed to know in order to optimally set the rbd_mirror_max_fetch_bytes parameter as it exists today.

Instead, it would be more practical to declare an upper limit on total amount of journal data that the daemon can retrieve per fetch, depending on network and memory parameters. This would avoid tuning this parameter frequently. Since the RBD-mirror daemon has prior knowledge of number of journal objects to be transmitted, it can then divide the value of this parameter by the number of journal objects needing at that point in time.