Project

General

Profile

Actions

Bug #50798

open

Memory leak ceph-mon octopus

Added by Александр Остапенко almost 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-deploy
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello!

Found a ceph-mon memory leak.
I attach the dumps and other additional information.

mon.ceph03 dumping heap profile now.
------------------------------------------------
MALLOC: 6767220984 ( 6453.7 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 115609616 ( 110.3 MiB) Bytes in central cache freelist
MALLOC: + 12262912 ( 11.7 MiB) Bytes in transfer cache freelist
MALLOC: + 25467128 ( 24.3 MiB) Bytes in thread cache freelists
MALLOC: + 49545216 ( 47.2 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 6970105856 ( 6647.2 MiB) Actual memory used (physical + swap)
MALLOC: + 7307264 ( 7.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 6977413120 ( 6654.2 MiB) Virtual address space used
MALLOC:
MALLOC: 829723 Spans in use
MALLOC: 24 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

ceph03:~# ceph -s
cluster:
id: 465e176e-1617-4575-824c-bd82f8a14d96
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph01,ceph02,ceph03 (age 15h)
mgr: ceph03(active, since 3w), standbys: ceph02
mds: 3 up:standby
osd: 6 osds: 6 up (since 2w), 6 in (since 4w)
rgw: 1 daemon active (d022i43dmh-ceph01)
task status:
data:
pools: 9 pools, 209 pgs
objects: 1.61k objects, 3.7 GiB
usage: 12 GiB used, 564 GiB / 576 GiB avail
pgs: 209 active+clean
io:
client: 27 KiB/s wr, 0 op/s rd, 3 op/s wr

ceph03:~# free -g
total used free shared buff/cache available
Mem: 7 7 0 0 0 0
Swap: 1 1 0

Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

ceph03:~# dpkg -l | grep ceph
ii ceph 15.2.11-1~bpo10+1 amd64 distributed storage and file system
ii ceph-base 15.2.11-1~bpo10+1 amd64 common ceph daemon libraries and management tools
ii ceph-common 15.2.11-1~bpo10+1 amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 15.2.8-1~bpo10+1 amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 15.2.11-1~bpo10+1 amd64 metadata server for the ceph distributed file system
ii ceph-mgr 15.2.11-1~bpo10+1 amd64 manager for the ceph distributed storage system
ii ceph-mgr-modules-core 15.2.11-1~bpo10+1 all ceph manager modules which are always enabled
ii ceph-mon 15.2.11-1~bpo10+1 amd64 monitor server for the ceph storage system
ii ceph-osd 15.2.11-1~bpo10+1 amd64 OSD server for the ceph storage system
rc cephadm 15.2.8-1~bpo10+1 amd64 cephadm utility to bootstrap ceph daemons with systemd and containers
ii libcephfs2 15.2.11-1~bpo10+1 amd64 Ceph distributed file system client library
ii python3-ceph-argparse 15.2.11-1~bpo10+1 all Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.11-1~bpo10+1 all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.11-1~bpo10+1 amd64 Python 3 libraries for the Ceph libcephfs library

After restarting ceph-mon, memory consumption drops, but gradually increases over time.
Any ideas?


Files

mon-ceph03.profile.0006.heap (440 KB) mon-ceph03.profile.0006.heap Александр Остапенко, 05/13/2021 07:49 AM
Actions #1

Updated by Loïc Dachary almost 3 years ago

  • Target version deleted (v15.2.11)
  • Affected Versions v15.2.11 added
Actions

Also available in: Atom PDF