Project

General

Profile

Bug #41009

osd_memory_target isn't applied in runtime.

Added by Igor Fedotov 8 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Looks like this PR (https://github.com/ceph/ceph/pull/27381) completely removed (intentionally?) the ability to adjust the parameter on the fly. If it had been ever enabled which is questionable IMO.

No FLAG_STARTUP for the option to notify user about that though.


Related issues

Related to bluestore - Bug #44010: changing osd_memory_target currently requires restart, should update at runtime New
Copied to bluestore - Backport #41810: nautilus: osd_memory_target isn't applied in runtime. Resolved

History

#1 Updated by Josh Durgin 8 months ago

  • Assignee set to Sridhar Seshasayee

Sridhar was planning on working on this after the mon_memory_target (https://github.com/ceph/ceph/pull/28227) - I hadn't realized it used to be injectable. The cache autotuning does need a config observer to be able to adjust things properly when relevant options change.

#2 Updated by Sridhar Seshasayee 8 months ago

  • Pull request ID set to 29606

PR https://github.com/ceph/ceph/pull/29606 is raised that addresses this issue.

Tested this using vstart and here are some logs of interest to confirm that the change works,

1. Change the default osd_memory_target to 2_G from 4_G.

MempoolThread picks up the value saved in bluestore object and updated the pcm,
58212 2019-08-12T16:13:35.547+0530 7f0d71334700  5 bluestore.MempoolThread(0x55f3f27dbd90) _update_cache_settings updated pcm target: 2147483648

Eventually the cache size is recalculated as part of _resize_shards shown below,

58274 2019-08-12T16:13:36.451+0530 7f0d71334700 20 bluestore.MempoolThread(0x55f3f27dbd90) _resize_shards cache_size: 1020054732 kv_alloc: 1073741824 kv_used: 1824 meta_alloc: 1073741824 me      ta_used: 124604 data_alloc: 671088640 data_used: 0

...
...

58467 2019-08-12T16:13:40.471+0530 7f0d71334700 10 rocksdb: commit_cache_size old: 1073741824 new: 390070272
58468 2019-08-12T16:13:40.471+0530 7f0d71334700 10 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.017348
58469 2019-08-12T16:13:40.471+0530 7f0d71334700  5 bluestore.MempoolThread(0x55f3f27dbd90) _resize_shards cache_size: 1020054732 kv_alloc: 390070272 kv_used: 1824 meta_alloc: 390070272 meta      _used: 124604 data_alloc: 230686720 data_used: 0

2. Pass an invalid osd_memory_target value of 0.

2019-08-12T18:35:10.141+0530 7ff5ff307700 -1 bluestore(/home/sseshasa/workspaces/ceph/build/dev/osd0) _update_osd_memory_target_size osd_memory_target:0. Invalid size provided.

As observed above, the invalid value is detected and bluestore object is not updated.

#3 Updated by Neha Ojha 8 months ago

  • Backport set to nautilus

#5 Updated by Sridhar Seshasayee 7 months ago

  • Status changed from New to Pending Backport

#6 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #41810: nautilus: osd_memory_target isn't applied in runtime. added

#8 Updated by Nathan Cutler 3 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

#9 Updated by Igor Fedotov about 2 months ago

  • Related to Bug #44010: changing osd_memory_target currently requires restart, should update at runtime added

Also available in: Atom PDF