Ceph

Ceph Kernel clients (both CephFS and RBD) has a MAX_FRONT_SIZE (16 MB) limit for ceph messages. If such clients received a MOSDMap message with front portion bigger than MAX_FRONT_SIZE, client would terminate the TCP connection and retry.

However, the retry will never succeed, as the cluster side map epoch increase, the requests will request more and more map versions, the response will be bigger and bigger. What is worse, the endless retries with requesting hundreds of versions at a time saturates CPU of monitor, and fill up its message queue.

Under certain circumstances (in combination of cluster size and diff between versions) the total size of N maps together with a few other metadata can exceed MAX_FRONT_SIZE, such message would trigger above mentioned endless retries then exhausted ceph-mon.

We hit this issue in our production environments, and seen other victim's report like https://www.spinics.net/lists/ceph-users/msg50441.html

The proposed fix is lets add a `osd_mon_messages_max_bytes` configuration, so that in `build_incremental` we can cap on both # of maps and # of bytes.