Project

General

Profile

Actions
Copy link

Bug #56093

open

log_max_recent setting broken as of Nautilus

Added by Joshua Baergen almost 2 years ago. Updated almost 2 years ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Joshua Baergen
Category:
common
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
octopus,pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v14.0.0, v14.1.0, v14.1.1, v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.12, v14.2.13, v14.2.14, v14.2.15, v14.2.16, v14.2.17, v14.2.18, v14.2.19, v14.2.2, v14.2.20, v14.2.21, v14.2.22, v14.2.23, v14.2.3, v14.2.4, v14.2.5, v14.2.6, v14.2.7, v14.2.8, v14.2.9, v15.0.0, v15.1.0, v15.1.1, v15.2.0, v15.2.1, v15.2.10, v15.2.11, v15.2.12, v15.2.13, v15.2.14, v15.2.15, v15.2.16, v15.2.2, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8, v15.2.9, v16.0.0, v16.0.1, v16.1.0, v16.1.1, v16.2.0, v16.2.1, v16.2.2, v16.2.3, v16.2.4, v16.2.5, v16.2.6, v16.2.7, v16.2.8, v17.0.0, v17.1.0, v18.0.0
ceph-qa-suite:
Pull request ID:
46736
Crash signature (v1):
Crash signature (v2):

Description

We have an internal monitoring tool that connects to several dozen Ceph clusters that was exhibiting rapidly growing memory usage after upgrading it to use Nautilus libraries rather than Luminous ones. Inspecting /proc/<pid>/smaps for growing memory regions, dumping them via gdb, and then inspecting them using 'strings' revealed that they were full of log messages. We turned most logging off and the memory usage was much better controlled.

Digging through the code, I found that https://github.com/ceph/ceph/commit/a747aeac13daf3dba43343120659e802cb569f6b broke the log_max_recent option, meaning that non-daemons no longer have a reduced in-memory recent log entry buffer, increasing the memory usage of that buffer 20x for our usecase. This is probably not all that noticeable when connecting to one or two clusters, but it's very noticeable when connecting to dozens!

I have a fix for this that I've successfully tested on Nautilus. PR coming soon.

Related issues 3 (1 open — 2 closed)

Copied to Ceph - Backport #56635: pacific: log_max_recent setting broken as of NautilusResolvedKonstantin ShalyginActions
Copied to Ceph - Backport #56636: octopus: log_max_recent setting broken as of NautilusRejectedActions
Copied to Ceph - Backport #56637: quincy: log_max_recent setting broken as of NautilusIn ProgressKonstantin ShalyginActions
Actions
Copy link

Also available in: Atom PDF