Ceph

We have an internal monitoring tool that connects to several dozen Ceph clusters that was exhibiting rapidly growing memory usage after upgrading it to use Nautilus libraries rather than Luminous ones. Inspecting /proc/<pid>/smaps for growing memory regions, dumping them via gdb, and then inspecting them using 'strings' revealed that they were full of log messages. We turned most logging off and the memory usage was much better controlled.

Digging through the code, I found that https://github.com/ceph/ceph/commit/a747aeac13daf3dba43343120659e802cb569f6b broke the log_max_recent option, meaning that non-daemons no longer have a reduced in-memory recent log entry buffer, increasing the memory usage of that buffer 20x for our usecase. This is probably not all that noticeable when connecting to one or two clusters, but it's very noticeable when connecting to dozens!

I have a fix for this that I've successfully tested on Nautilus. PR coming soon.