Project

General

Profile

Actions

Bug #55760

open

ceph osd bluestore memory leak

Added by dovefi Z almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
osd memory leak
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Problem
hi, everyone, we have a ceph cluster, and we only use rgw with EC Pool, now the cluster osd memory keeps growing to 16GB

  • ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
  • osd config
    ceph daemon osd.374 config get osd_memory_target
    {
        "osd_memory_target": "4294967296" 
    }
    

osd used memory
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
2158658 ceph      20   0 17.023g 0.016t  10700 S   5.6 12.9  97:37.84 ceph-osd
2167500 ceph      20   0 16.503g 0.015t  17392 S   1.7 12.1 103:05.32 ceph-osd
2164715 ceph      20   0 14.230g 0.013t  19904 S   1.3 10.6 101:39.31 ceph-osd
2163249 ceph      20   0 13.249g 0.012t  20068 S   1.3  9.9  94:41.75 ceph-osd
2160621 ceph      20   0 12.728g 0.012t   9256 S   2.3  9.4 103:20.76 ceph-osd
2158042 ceph      20   0 12.748g 0.011t  22344 S   1.0  9.3  88:23.16 ceph-osd
2155989 ceph      20   0 11.843g 0.010t  22656 S   4.3  8.5  91:13.34 ceph-osd
2168721 ceph      20   0 10.896g 0.010t  20204 S   4.6  8.0  93:46.26 ceph-osd
2166929 ceph      20   0 10.458g 9.538g  11004 S   1.3  7.6  91:26.08 ceph-osd
2161262 ceph      20   0 9011516 7.734g   9600 S   1.7  6.2  98:06.58 ceph-osd

osd heap dump
osd.374 dumping heap profile now.
------------------------------------------------
MALLOC:    16957628224 (16172.1 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    241987208 (  230.8 MiB) Bytes in central cache freelist
MALLOC: +     12445504 (   11.9 MiB) Bytes in transfer cache freelist
MALLOC: +     81128184 (   77.4 MiB) Bytes in thread cache freelists
MALLOC: +     75378880 (   71.9 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =  17368568000 (16564.0 MiB) Actual memory used (physical + swap)
MALLOC: +      3424256 (    3.3 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =  17371992256 (16567.2 MiB) Virtual address space used
MALLOC:
MALLOC:        1110700              Spans in use
MALLOC:             47              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

osd used memory 16GB

osd dump_mempools
{
    "bloom_filter": {
        "items": 0,
        "bytes": 0
    },
    "bluestore_alloc": {
        "items": 15718400,
        "bytes": 15718400
    },
    "bluestore_cache_data": {
        "items": 0,
        "bytes": 0
    },
    "bluestore_cache_onode": {
        "items": 2747609,
        "bytes": 1846393248
    },
    "bluestore_cache_other": {
        "items": 682869874,
        "bytes": 9725851995
    },
    "bluestore_fsck": {
        "items": 0,
        "bytes": 0
    },
    "bluestore_txc": {
        "items": 60,
        "bytes": 44160
    },
    "bluestore_writing_deferred": {
        "items": 641,
        "bytes": 2639419
    },
    "bluestore_writing": {
        "items": 1585,
        "bytes": 6707995
    },
    "bluefs": {
        "items": 5971,
        "bytes": 227232
    },
    "buffer_anon": {
        "items": 16814,
        "bytes": 15956676
    },
    "buffer_meta": {
        "items": 115,
        "bytes": 10120
    },
    "osd": {
        "items": 117,
        "bytes": 1480752
    },
    "osd_mapbl": {
        "items": 0,
        "bytes": 0
    },
    "osd_pglog": {
        "items": 525725,
        "bytes": 103983300
    },
    "osdmap": {
        "items": 137722,
        "bytes": 3183664
    },
    "osdmap_mapping": {
        "items": 0,
        "bytes": 0
    },
    "pgmap": {
        "items": 0,
        "bytes": 0
    },
    "mds_co": {
        "items": 0,
        "bytes": 0
    },
    "unittest_1": {
        "items": 0,
        "bytes": 0
    },
    "unittest_2": {
        "items": 0,
        "bytes": 0
    },
    "total": {
        "items": 702024633,
        "bytes": 11722196961
    }
}

the bluestore_cache_other value is very high

how can we do to fix this bug??

No data to display

Actions

Also available in: Atom PDF