Project

General

Profile

Bug #58281

osd:memory usage exceeds the osd_memory_target

Added by yite gu over 1 year ago. Updated over 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I want to limit osd_memory_target to 3758096384 bytes

# ceph tell osd.1 heap stats
osd.1 tcmalloc heap stats:------------------------------------------------
MALLOC:     3307415472 ( 3154.2 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +    692482368 (  660.4 MiB) Bytes in central cache freelist
MALLOC: +      4328960 (    4.1 MiB) Bytes in transfer cache freelist
MALLOC: +     27154704 (   25.9 MiB) Bytes in thread cache freelists
MALLOC: +     26083328 (   24.9 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =   4057464832 ( 3869.5 MiB) Actual memory used (physical + swap)
MALLOC: +    164577280 (  157.0 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =   4222042112 ( 4026.5 MiB) Virtual address space used
MALLOC:
MALLOC:         380168              Spans in use
MALLOC:             33              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

# ceph tell osd.1 config get osd_memory_target
{
    "osd_memory_target": "3758096384" 
}

# kubectl top pod -n rook-ceph | grep ceph-osd-1-
NAME                                                      CPU(cores)   MEMORY(bytes)
rook-ceph-osd-1-85cfd4889d-f6wnh                          17m          3930Mi

History

#1 Updated by yite gu over 1 year ago

# ceph daemon osd.1 dump_mempools
{
    "mempool": {
        "by_pool": {
            "bloom_filter": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_alloc": {
                "items": 4669668,
                "bytes": 97787664
            },
            "bluestore_cache_data": {
                "items": 53,
                "bytes": 356352
            },
            "bluestore_cache_onode": {
                "items": 187393,
                "bytes": 115434088
            },
            "bluestore_cache_other": {
                "items": 26003584,
                "bytes": 1378358683
            },
            "bluestore_fsck": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_txc": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_writing_deferred": {
                "items": 14,
                "bytes": 57344
            },
            "bluestore_writing": {
                "items": 0,
                "bytes": 0
            },
            "bluefs": {
                "items": 1385,
                "bytes": 34288
            },
            "buffer_anon": {
                "items": 186125,
                "bytes": 52922262
            },
            "buffer_meta": {
                "items": 185901,
                "bytes": 16359288
            },
            "osd": {
                "items": 32,
                "bytes": 413952
            },
            "osd_mapbl": {
                "items": 33,
                "bytes": 383658
            },
            "osd_pglog": {
                "items": 308575,
                "bytes": 160459000
            },
            "osdmap": {
                "items": 4933,
                "bytes": 298024
            },
            "osdmap_mapping": {
                "items": 0,
                "bytes": 0
            },
            "pgmap": {
                "items": 0,
                "bytes": 0
            },
            "mds_co": {
                "items": 0,
                "bytes": 0
            },
            "unittest_1": {
                "items": 0,
                "bytes": 0
            },
            "unittest_2": {
                "items": 0,
                "bytes": 0
            }
        },
        "total": {
            "items": 31547696,
            "bytes": 1822864603
        }
    }
}

#2 Updated by yite gu over 1 year ago

ceph daemon osd.1 perf dump

    "prioritycache": {
        "target_bytes": 3758096384,
        "mapped_bytes": 4032946176,
        "unmapped_bytes": 163012608,
        "heap_bytes": 4195958784,
        "cache_bytes": 134217728
    },

#3 Updated by Igor Fedotov over 1 year ago

Please note that osd_memory_target is not a hard limit. It's just 'target' OSD usage that OSD attempts to align with.

Additionally Ceph documentation provides the following notes regarding osd_memory_target:
"
When TCMalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should usually be close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial development, it was found that some kernels result in the OSD’s RSS memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary.
"

#4 Updated by yite gu over 1 year ago

Igor Fedotov wrote:

Please note that osd_memory_target is not a hard limit. It's just 'target' OSD usage that OSD attempts to align with.

Additionally Ceph documentation provides the following notes regarding osd_memory_target:
"
When TCMalloc is available and cache autotuning is enabled, try to keep this many bytes mapped in memory. Note: This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should usually be close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped. During initial development, it was found that some kernels result in the OSD’s RSS memory exceeding the mapped memory by up to 20%. It is hypothesised however, that the kernel generally may be more aggressive about reclaiming unmapped memory when there is a high amount of memory pressure. Your mileage may vary.
"

thx,I got it :)

#5 Updated by Igor Fedotov over 1 year ago

  • Status changed from New to Rejected

Also available in: Atom PDF