Ceph » mgr

This is an inevitable question，I did the following operations.
1.Collects heap information
ceph tell mgr.HOSTNAME heap start_profiler （Not supported by default, I added it myself）
ceph tell mgr.HOSTNAME heap dump
2.Processes heap files
pprof --pdf /home/zhangbo/ceph-mgr /var/log/ceph/mgr.node149.profile.0002.heap >test.heap.pdf
See the attachment for the test.heap.pdf
3.Compare the data from two samplings, with an interval of about 2 hours.
pprof --pdf --base /var/log/ceph/mgr.node149.profile.0004.heap /home/zhangbo/ceph-mgr /var/log/ceph/mgr.node149.profile.0050.heap > test.compare.pdf
See the attachment for the test.compare.pdf

I tested the above on 14.2.8, and 17.2.5 also has this problem,And more serious.
The memory of ceph-mgr continues to increase and there is no downward trend. I noticed that the memory of ceph-mgr has increased to 20G or even higher.If the mgr service has not been restarted for several years, the memory may also gradually increase (uses the default log level 1/5).

The ceph-mon has the same problem, but the log volume of mon is less than that of mgr (debug_mon = debug_mgr =30/30). The phenomenon is that the memory of ceph-mon increases more slowly and takes up less space.

This bug has the potential to OOM the whole system.