Feature #44011
openallow setting osd_memory_target per device class
0%
Description
We are using a deployment where a flat osd_memory_target is not suitable. It would be very helpful if it was possible to assign a memory target based on device class, for example, with a command like
ceph config set class.ssd osd_memory_target 8589934592
Reasons for this feature:
- When deploying OSDs in an automated way (auto-discovery, random order) it is not possible to assign individual memory-targets a-priory. Using device-class settings would allow at least some control over initial assignments and reduce the amount of post-deployment tasks.
- If one uses several different memory targets for large groups of OSDs, the output of "ceph config dump" gets unnecessarily polluted with settings that would naturally apply to entire device classes.
Changing a device class should update this setting at run-time, see also https://tracker.ceph.com/issues/44010 .
Updated by Sridhar Seshasayee about 4 years ago
- Project changed from RADOS to Ceph
You can set a configuration value for a device class using:
ceph config set osd/class:ssd <option> <value>
For e.g., to set osd_memory_target for osd.0 with a device class ssd one can use:
ceph config set osd.0/class:ssd osd_memory_target <value>
Bear in mind "osd_memory_target" is not configurable at runtime in mimic but
from nautilus and beyond.
Updated by Frank Schilder about 4 years ago
Oh dear, a classic RTFM case. I found it in the documentation now.
I have three questions:
1) Would it be possible to add this sentence to the description of osd_memory_target in section "Automatic Cache Sizing" of bluestore config:
To set different values depending on device class or crush location, [please use masks](link to Sections and masks in configuring ceph).
2) The rules of precedence are not clear from the docs. If I set a value for "osd/host:host-01" and another value for "osd/class:ssd" and I have an SSD on host-01, which value will win? I would expect the "class" spec, because on host-01 the class SSD is more special than any class. Does this match the implementation?
3) Regarding point (2), is it possible to combine masks as in "osd/host:host-01,class:ssd" (combination is AND to have strictly increasing specialization). I'm asking because I have the situation that I would like to set two different default values for SSDs for two different crush roots, that is, I would really love to be able to set something like "osd/datacenter:location1,class:ssd" and "osd/datacenter:location2,class:ssd" to different values.
To avoid complicated disambiguation rules, one could say that "class" always wins over "host", which would imply a strict tree structure on these settings with simple disambiguation rules.
Updated by Frank Schilder about 4 years ago
Update: I tried setting
ceph config set osd/class:hdd osd_memory_target 2147483648
as documented and restarted an HDD backed OSD. Unfortunately, the value (2G) is not used, but rather the default value (4G).
See also results of my experiments here: https://tracker.ceph.com/issues/44010#note-8.