Setting values depending on device class is described here: https://docs.ceph.com/docs/mimic/rados/configuration/ceph-conf/#sections-and-masks .
However, it seems not to work. Here the steps I took:
1. Start OSDs with default value for osd_memory_target (4G). All OSDs start consuming close to 4G as expected.
2. Set osd_memory_target to 2G and wait. Expected behaviour is that OSDs start reducing their consumption. This does not work and was the reason for this ticket.
3. Restart all OSDs. Now the new value for osd_memory_target takes effect and OSDs stay below 2G.
Now I tried with the device class setting using filters, but this seems not to work as intended.
4. Remove the osd_memory_target setting from step 2 with
ceph config rm osd osd_memory_target
5. Set class-specific target:
ceph config set osd/class:hdd osd_memory_target 2147483648
6. A config dump now contains this line:
WHO MASK LEVEL OPTION VALUE RO
osd class:hdd basic osd_memory_target 2147483648
7. Restart an OSD backed by a HDD. Expected behaviour is that the OSD stays below 2G as all the others. Unfortunately, this does
not work as documented (output from top):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
785197 ceph 20 0 5672756 3.9g 9564 S 0.0 6.3 236:23.47 /usr/bin/ceph-osd --cluster ceph -f -i 34 --setuser ceph --setgro+
979905 ceph 20 0 4355268 3.0g 22952 S 1.0 4.7 4:25.64 /usr/bin/ceph-osd --cluster ceph -f -i 227 --setuser ceph --setgr+
898846 ceph 20 0 2996032 1.9g 8988 S 1.0 3.0 173:10.26 /usr/bin/ceph-osd --cluster ceph -f -i 96 --setuser ceph --setgro+
900352 ceph 20 0 2948024 1.9g 9008 S 1.7 3.0 166:09.68 /usr/bin/ceph-osd --cluster ceph -f -i 206 --setuser ceph --setgr+
897236 ceph 20 0 2892108 1.8g 8464 S 1.0 2.9 149:46.43 /usr/bin/ceph-osd --cluster ceph -f -i 225 --setuser ceph --setgr+
895713 ceph 20 0 3037724 1.8g 8692 S 1.3 2.9 144:59.47 /usr/bin/ceph-osd --cluster ceph -f -i 198 --setuser ceph --setgr+
899216 ceph 20 0 3001248 1.8g 8652 S 1.3 2.9 138:01.32 /usr/bin/ceph-osd --cluster ceph -f -i 200 --setuser ceph --setgr+
894927 ceph 20 0 2973764 1.8g 8804 S 0.7 2.8 163:52.28 /usr/bin/ceph-osd --cluster ceph -f -i 194 --setuser ceph --setgr+
896893 ceph 20 0 3004720 1.8g 8524 S 1.0 2.8 178:03.53 /usr/bin/ceph-osd --cluster ceph -f -i 222 --setuser ceph --setgr+
899659 ceph 20 0 2970016 1.7g 9332 S 0.7 2.8 150:56.23 /usr/bin/ceph-osd --cluster ceph -f -i 202 --setuser ceph --setgr+
897651 ceph 20 0 2934408 1.7g 8644 S 3.7 2.7 161:30.68 /usr/bin/ceph-osd --cluster ceph -f -i 133 --setuser ceph --setgr+
895245 ceph 20 0 2952512 1.7g 9096 S 1.0 2.7 204:53.27 /usr/bin/ceph-osd --cluster ceph -f -i 196 --setuser ceph --setgr+
898033 ceph 20 0 2939056 1.7g 9056 S 1.0 2.6 160:57.36 /usr/bin/ceph-osd --cluster ceph -f -i 120 --setuser ceph --setgr+
898441 ceph 20 0 2906020 1.6g 8264 S 1.3 2.6 149:21.77 /usr/bin/ceph-osd --cluster ceph -f -i 109 --setuser ceph --setgr+
899995 ceph 20 0 2947384 1.6g 9164 S 0.7 2.6 143:57.95 /usr/bin/ceph-osd --cluster ceph -f -i 204 --setuser ceph --setgr+
896046 ceph 20 0 2972096 1.6g 9076 S 1.3 2.6 181:17.36 /usr/bin/ceph-osd --cluster ceph -f -i 224 --setuser ceph --setgr+
OSD 34 is on SSD, so the 3.9G is OK. OSD 227 is the one on HDD I restarted in step 7. It does not use the value set with the class:hdd mask. All other OSDs were restarted in step 2 with osd_memory_target set to 2G.
Here is the crush tree for this host:
-136 134.01401 host ceph-16
96 hdd 8.90999 osd.96
109 hdd 8.90999 osd.109
120 hdd 8.90999 osd.120
133 hdd 8.90999 osd.133
194 hdd 8.90999 osd.194
196 hdd 8.90999 osd.196
198 hdd 8.90999 osd.198
200 hdd 8.90999 osd.200
202 hdd 8.90999 osd.202
204 hdd 8.90999 osd.204
206 hdd 8.90999 osd.206
222 hdd 8.90999 osd.222
224 hdd 8.90999 osd.224
225 hdd 8.90999 osd.225
227 hdd 8.90999 osd.227
34 ssd 0.36400 osd.34
Now it looks like there are actually two bugs here:
- Changing osd_memory_target requires a restart but should not.
- Setting config values with masks does not work at all, restart or not.
I don't use nautilus, so I can't comment on the behaviour there. For mimic it seems to work if OSDs get the correct value. As I wrote above, priority has getting config values with masks working as documented. I personally can live with the restarts if masks work properly.