Bug #58281
osd:memory usage exceeds the osd_memory_target
0%
Description
I want to limit osd_memory_target to 3758096384 bytes
# ceph tell osd.1 heap stats osd.1 tcmalloc heap stats:------------------------------------------------ MALLOC: 3307415472 ( 3154.2 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 692482368 ( 660.4 MiB) Bytes in central cache freelist MALLOC: + 4328960 ( 4.1 MiB) Bytes in transfer cache freelist MALLOC: + 27154704 ( 25.9 MiB) Bytes in thread cache freelists MALLOC: + 26083328 ( 24.9 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 4057464832 ( 3869.5 MiB) Actual memory used (physical + swap) MALLOC: + 164577280 ( 157.0 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4222042112 ( 4026.5 MiB) Virtual address space used MALLOC: MALLOC: 380168 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.
# ceph tell osd.1 config get osd_memory_target
{
"osd_memory_target": "3758096384"
}
# kubectl top pod -n rook-ceph | grep ceph-osd-1- NAME CPU(cores) MEMORY(bytes) rook-ceph-osd-1-85cfd4889d-f6wnh 17m 3930Mi
History
#1 Updated by yite gu 12 months ago
# ceph daemon osd.1 dump_mempools
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 0,
"bytes": 0
},
"bluestore_alloc": {
"items": 4669668,
"bytes": 97787664
},
"bluestore_cache_data": {
"items": 53,
"bytes": 356352
},
"bluestore_cache_onode": {
"items": 187393,
"bytes": 115434088
},
"bluestore_cache_other": {
"items": 26003584,
"bytes": 1378358683
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 0,
"bytes": 0
},
"bluestore_writing_deferred": {
"items": 14,
"bytes": 57344
},
"bluestore_writing": {
"items": 0,
"bytes": 0
},
"bluefs": {
"items": 1385,
"bytes": 34288
},
"buffer_anon": {
"items": 186125,
"bytes": 52922262
},
"buffer_meta": {
"items": 185901,
"bytes": 16359288
},
"osd": {
"items": 32,
"bytes": 413952
},
"osd_mapbl": {
"items": 33,
"bytes": 383658
},
"osd_pglog": {
"items": 308575,
"bytes": 160459000
},
"osdmap": {
"items": 4933,
"bytes": 298024
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 0,
"bytes": 0
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 31547696,
"bytes": 1822864603
}
}
}
#3 Updated by Igor Fedotov 12 months ago
Please note that osd_memory_target is not a hard limit. It's just 'target' OSD usage that OSD attempts to align with.
Additionally Ceph documentation provides the following notes regarding osd_memory_target:
"
When TCMalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should usually be close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial development, it was found that some kernels result in the OSD’s RSS memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary.
"
#4 Updated by yite gu 12 months ago
Igor Fedotov wrote:
Please note that osd_memory_target is not a hard limit. It's just 'target' OSD usage that OSD attempts to align with.
Additionally Ceph documentation provides the following notes regarding osd_memory_target:
"
When TCMalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should usually be close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial development, it was found that some kernels result in the OSD’s RSS memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary.
"
thx，I got it :)
#5 Updated by Igor Fedotov 12 months ago
- Status changed from New to Rejected