Bug #56093
open
log_max_recent setting broken as of Nautilus
Added by Joshua Baergen almost 2 years ago.
Updated almost 2 years ago.
Backport:
octopus,pacific,quincy
Affected Versions:
v14.0.0,
v14.1.0,
v14.1.1,
v14.2.0,
v14.2.1,
v14.2.10,
v14.2.11,
v14.2.12,
v14.2.13,
v14.2.14,
v14.2.15,
v14.2.16,
v14.2.17,
v14.2.18,
v14.2.19,
v14.2.2,
v14.2.20,
v14.2.21,
v14.2.22,
v14.2.23,
v14.2.3,
v14.2.4,
v14.2.5,
v14.2.6,
v14.2.7,
v14.2.8,
v14.2.9,
v15.0.0,
v15.1.0,
v15.1.1,
v15.2.0,
v15.2.1,
v15.2.10,
v15.2.11,
v15.2.12,
v15.2.13,
v15.2.14,
v15.2.15,
v15.2.16,
v15.2.2,
v15.2.3,
v15.2.4,
v15.2.5,
v15.2.6,
v15.2.7,
v15.2.8,
v15.2.9,
v16.0.0,
v16.0.1,
v16.1.0,
v16.1.1,
v16.2.0,
v16.2.1,
v16.2.2,
v16.2.3,
v16.2.4,
v16.2.5,
v16.2.6,
v16.2.7,
v16.2.8,
v17.0.0,
v17.1.0,
v18.0.0
Description
We have an internal monitoring tool that connects to several dozen Ceph clusters that was exhibiting rapidly growing memory usage after upgrading it to use Nautilus libraries rather than Luminous ones. Inspecting /proc/<pid>/smaps for growing memory regions, dumping them via gdb, and then inspecting them using 'strings' revealed that they were full of log messages. We turned most logging off and the memory usage was much better controlled.
Digging through the code, I found that https://github.com/ceph/ceph/commit/a747aeac13daf3dba43343120659e802cb569f6b broke the log_max_recent option, meaning that non-daemons no longer have a reduced in-memory recent log entry buffer, increasing the memory usage of that buffer 20x for our usecase. This is probably not all that noticeable when connecting to one or two clusters, but it's very noticeable when connecting to dozens!
I have a fix for this that I've successfully tested on Nautilus. PR coming soon.
- Status changed from New to Fix Under Review
- Assignee set to Joshua Baergen
- Pull request ID set to 46736
- Backport set to octopus,pacific,quincy
- Status changed from Fix Under Review to Pending Backport
- Copied to Backport #56635: pacific: log_max_recent setting broken as of Nautilus added
- Copied to Backport #56636: octopus: log_max_recent setting broken as of Nautilus added
- Copied to Backport #56637: quincy: log_max_recent setting broken as of Nautilus added
- Tags set to backport_processed
Also available in: Atom
PDF