Project

General

Profile

Actions

Bug #64078

open

When I set debug_mgr 30/30, the memory of ceph-mgr will keep rising

Added by Bo Zhang 4 months ago. Updated 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is an inevitable question,I did the following operations.
1.Collects heap information
ceph tell mgr.HOSTNAME heap start_profiler (Not supported by default, I added it myself)
ceph tell mgr.HOSTNAME heap dump
2.Processes heap files
pprof --pdf /home/zhangbo/ceph-mgr /var/log/ceph/mgr.node149.profile.0002.heap >test.heap.pdf

See the attachment for the test.heap.pdf


Files

test.heap.pdf (15.5 KB) test.heap.pdf Bo Zhang, 01/18/2024 05:58 AM
test.compare.pdf (13.3 KB) test.compare.pdf Bo Zhang, 01/18/2024 06:35 AM

Related issues 1 (1 open0 closed)

Has duplicate mgr - Bug #64082: When I set debug_mgr 30/30, the memory of ceph-mgr will keep risingNew

Actions
Actions #1

Updated by Bo Zhang 4 months ago

The memory of ceph-mgr continues to increase and there is no downward trend. I noticed that the memory of ceph-mgr has increased to 20G or even higher.

This bug has the potential to OOM the whole system.

Actions #2

Updated by Bo Zhang 4 months ago

If the mgr service has not been restarted for several years, the memory may also gradually increase (uses the default log level 1/5)

Actions #3

Updated by Bo Zhang 4 months ago

Compare the data from two samplings, with an interval of about 2 hours.

pprof --pdf --base /var/log/ceph/mgr.node149.profile.0004.heap /home/zhangbo/ceph-mgr /var/log/ceph/mgr.node149.profile.0050.heap > test.compare.pdf

See the attachment for the test.compare.pdf

Actions #4

Updated by Igor Fedotov 4 months ago

  • Project changed from bluestore to mgr
Actions #5

Updated by Igor Fedotov 4 months ago

  • Has duplicate Bug #64082: When I set debug_mgr 30/30, the memory of ceph-mgr will keep rising added
Actions

Also available in: Atom PDF