Project

General

Profile

Bug #46125

ceph mon memory increasing

Added by Ashish Nagar over 3 years ago. Updated over 3 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi,

I have deployed ceph single node cluster.

ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)

ceph mon memory is continuously increasing.

1. Is there any configuration to limit mon memory?

2. Are there any kernel level settings? Using Debian buster.

3. Is this related to any logging / caching which can be limited by some configuration settings?

Thanks for help in advance.

History

#1 Updated by Ashish Nagar over 3 years ago

Hi,

I have deployed ceph single node cluster.

ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)

ceph mon memory is continuously increasing.

1. Is there any configuration to limit mon memory?

2. Are there any kernel level settings? Using Debian buster.

3. Is this related to any logging / caching which can be limited by some configuration settings?

Thanks for help in advance.

ceph.conf (for reference):

[global]
  1. Cluster filesystem ID.
    fsid = 71d5dea6-2e29-451f-a356-df330e0ef673
  1. MDS related configurations.
    mds_session_blacklist_on_evict = false
    mds_session_blacklist_on_timeout = false
    mds session autoclose = 30
  1. The newly created image will only have layering enabled.
    rbd_default_features = 1
  1. Monitor related configurations.
    mon osd full ratio = .95
    mon sync trim timeout = 30.0
    mon sync heartbeat timeout = 30.0
    mon sync heartbeat interval = 5.0
    mon sync timeout = 30.0
    mon sync max retries = 5
    mon sync max payload size = 1045676
    mon accept timeout = 10.0
    mon client hunt interval = 3.0
    mon client ping interval = 10.0
    mon allow pool delete = true
    mon pg warn min per osd = 0
    mon pg warn max per osd = 32768
    mon pg warn max object skew = 10000
    mon clock drift allowed = 5
  1. Ceph Auth related configurations.
    auth cluster required = cephx
    auth service required = cephx
    auth client required = cephx
  1. OSD related configurations.
    osd journal size = 1024
    osd pool default size = 1
    osd pool default min size = 1
    osd pool default pg num = 4
    osd pool default pgp num = 4
    osd crush chooseleaf type = 0
  1. Maximum OSD allowed.
    mon max osd = 10000
  1. Enable the bluestore as a object storage backend.
    osd objectstore = bluestore
  1. Disable bind to MSGR-V2 protocol.
    ms bind msgr2 = false
  1. Set memory for OSD.
    osd_memory_target = 1073741824
  1. Set memory for MON and MGR.
    mon_osd_cache_size = 10
    rocksdb_cache_size = 1073741824
    mon_osd_cache_size_min = 134217728

osd_min_pg_log_entries = 10
osd_max_pg_log_entries = 10
osd_pg_log_dups_tracked = 10
osd_pg_log_trim_min = 10

bluestore_cache_autotune = 0
bluestore_cache_kv_ratio = 0.2
bluestore_cache_meta_ratio = 0.8

bluestore_rocksdb_options = compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB

ms bind ipv6 = false
public network = 10.30.0.0/16
cluster network = 10.30.0.0/16
mon initial members = px-aaee7abfa2fa

[mon.px-aaee7abfa2fa]
host = px-aaee7abfa2fa
mon addr = 10.30.0.154:6789,10.30.0.154:6789

[mds.px-aaee7abfa2fa]
host = px-aaee7abfa2fa

[osd.0]
host = px-aaee7abfa2fa
public addr = 10.30.0.154
cluster addr = 10.30.0.154
devs = /dev/sdb

#2 Updated by Josh Durgin over 3 years ago

  • Status changed from New to Need More Info

Can you try with the default settings for the monitor? What level of memory usage are you seeing exactly?

There is a mon_memory_target option, which controls how much memory the monitor uses by adjusting the osdmap cache. It may not help if the memory is used elsewhere though.

#3 Updated by Ashish Nagar over 3 years ago

I will try with default settings for the monitor. With current config file parameters, the monitor is using 1GB.

I have not used mon_memory_target. The default value of this parameters is 2 GB. can I use low value like 256 MB? What is the correct value to to set for mon_memory_target?

#4 Updated by Josh Durgin over 3 years ago

You can attempt to use a lower target, it's not something we've tested much for the monitors. We expect the monitor to keep within mon_memory_target - since you only saw 1G of a 2G target used, this does not sound like a bug.

Also available in: Atom PDF