Ceph » bluestore

All of my clients are CephFS FWIW

Richard, you have the following line in your config:
osd.* dev bluestore_cache_autotune false

Which disables cache autotuning, is that correct?

Igor Fedotov wrote:

Richard, you have the following line in your config:
osd.* dev bluestore_cache_autotune false

Which disables cache autotuning, is that correct?

Correct. I tried working the osd_memory_target down bit by bit, but it had no effect. Believing that the auto-tuning had a defect, I disabled it. The OSD should then use the default cache size value:

$ ceph daemon osd.80 config get bluestore_cache_size_hdd
{
    "bluestore_cache_size_hdd": "1073741824" 
}

I realize that OSD process will use memory for things other than cache, but they're running about 3.5-4GB of RSS now (and still getting OOM killed).

Hi Richard,

Sorry for the long latency on this reply! Setting the osd_memory_target won't do anything if you disable autotuning, so you will be back to setting the cache size and dividing it up using the various ratios. If you are still having this problem, you can try to diagnose what the autotuner is doing by setting debug_bluestore = 5 and debug_prioritycache = 5 and looking for lines that look roughly like this in the osd log:

2019-07-18T19:34:42.587-0400 7f4048b8d700  5 prioritycache tune_memory target: 4294967296 mapped: 4260962304 unmapped: 856948736 heap: 5117911040 old mem: 2845415707 new mem: 2845415707
2019-07-18T19:34:33.527-0400 7f4048b8d700  5 bluestore.MempoolThread(0x55a6d330ead0) _resize_shards cache_size: 2845415707 kv_alloc: 1241513984 kv_used: 874833889 meta_alloc: 1258291200 meta_used: 889040246 data_alloc: 318767104 data_used: 0

The first line will tell you what your memory target is set to, how much memory is currently mapped, how much is unmapped (ie what's been freed but the kernel hasn't reclaimed), the total heap size, and the old and new aggregate size for all of bluestores caches. The second line also tells you the aggregate cache size, and then how much space is being allocated and used for the kv, meta, and data caches. If there's a leak somewhere in the OSD or bluestore the autotuner will shrink the cache way down but eventually won't be able to contain it and eventually your process will start growing beyond the target size despite having a tiny amount of bluestore cache. If it's something else like a huge amount of freed memory not being reclaimed by the kernel, you'll see large amount of unmapped memory and a big heap size despite the mapped memory staying near the target. If it's a bug in the autotuner, we might see the mapped memory greatly exceeding the target.

Mark

I'm no longer running nautilus (or trying to run it), so I can't get more additional information. I was just reporting the issue. We had auto-tuning disabled and had "bluestore_cache_size_hdd": "1073741824" in the config. That should keep things fairly stable around the 1073741824 mark per OSD, yeah? Maybe even see 20-40% higher than that value, but I wouldn't expect to see it go 300% above.

We're fairly stable on Luminous again with this cluster, so I don't think Nautilus will be in the cards for some time.

I guess you can close this bug if there's nothing obvious jumping out in the code. I just wanted to report it.

I'm not sure why, but enabling the balancer module has solved the memory bug. Everything got smooth and stable once balancer ran (in crush compat mode).