Actions
Bug #55760
openceph osd bluestore memory leak
Status:
New
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
osd memory leak
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Problem
hi, everyone, we have a ceph cluster, and we only use rgw with EC Pool, now the cluster osd memory keeps growing to 16GB¶
- ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
- osd config
ceph daemon osd.374 config get osd_memory_target { "osd_memory_target": "4294967296" }
osd used memory
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2158658 ceph 20 0 17.023g 0.016t 10700 S 5.6 12.9 97:37.84 ceph-osd
2167500 ceph 20 0 16.503g 0.015t 17392 S 1.7 12.1 103:05.32 ceph-osd
2164715 ceph 20 0 14.230g 0.013t 19904 S 1.3 10.6 101:39.31 ceph-osd
2163249 ceph 20 0 13.249g 0.012t 20068 S 1.3 9.9 94:41.75 ceph-osd
2160621 ceph 20 0 12.728g 0.012t 9256 S 2.3 9.4 103:20.76 ceph-osd
2158042 ceph 20 0 12.748g 0.011t 22344 S 1.0 9.3 88:23.16 ceph-osd
2155989 ceph 20 0 11.843g 0.010t 22656 S 4.3 8.5 91:13.34 ceph-osd
2168721 ceph 20 0 10.896g 0.010t 20204 S 4.6 8.0 93:46.26 ceph-osd
2166929 ceph 20 0 10.458g 9.538g 11004 S 1.3 7.6 91:26.08 ceph-osd
2161262 ceph 20 0 9011516 7.734g 9600 S 1.7 6.2 98:06.58 ceph-osd
osd heap dump
osd.374 dumping heap profile now.
------------------------------------------------
MALLOC: 16957628224 (16172.1 MiB) Bytes in use by application
MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
MALLOC: + 241987208 ( 230.8 MiB) Bytes in central cache freelist
MALLOC: + 12445504 ( 11.9 MiB) Bytes in transfer cache freelist
MALLOC: + 81128184 ( 77.4 MiB) Bytes in thread cache freelists
MALLOC: + 75378880 ( 71.9 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 17368568000 (16564.0 MiB) Actual memory used (physical + swap)
MALLOC: + 3424256 ( 3.3 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 17371992256 (16567.2 MiB) Virtual address space used
MALLOC:
MALLOC: 1110700 Spans in use
MALLOC: 47 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
osd used memory 16GB
osd dump_mempools
{
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 15718400,
"bytes": 15718400
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 2747609,
"bytes": 1846393248
},
"bluestore_cache_other": {
"items": 682869874,
"bytes": 9725851995
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 60,
"bytes": 44160
},
"bluestore_writing_deferred": {
"items": 641,
"bytes": 2639419
},
"bluestore_writing": {
"items": 1585,
"bytes": 6707995
},
"bluefs": {
"items": 5971,
"bytes": 227232
},
"buffer_anon": {
"items": 16814,
"bytes": 15956676
},
"buffer_meta": {
"items": 115,
"bytes": 10120
},
"osd": {
"items": 117,
"bytes": 1480752
},
"osd_mapbl": {
"items": 0,
"bytes": 0
},
"osd_pglog": {
"items": 525725,
"bytes": 103983300
},
"osdmap": {
"items": 137722,
"bytes": 3183664
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
},
"total": {
"items": 702024633,
"bytes": 11722196961
}
}
the bluestore_cache_other value is very high
how can we do to fix this bug??
No data to display
Actions