Bug #20385
closedjemalloc+Bluestore+BlueFS causes unexpected RSS Memory usage growth
0%
Description
When testing standalone BlueStore FIO plugin one can observe excessive RSS memory usage for corresponding FIO process. This occurs in case of write scenarios and BlueFS use only. Regular OSD probably suffers from the same issue as well.
The root cause is the lack for BlueFS::_flush_bdev_safely() call for one of BlueFS files that probably results in memory fragmentation and prevents from returning unused pages to OS. I suppose the file in question is log_writer one. In fact that's rather lack of BlueFS::flush_log() call that's absent over the code.
One can insert corresponding function call (_flush_bdev_safely()) at the end of BlueFS::_flush_range as a workaround and/or root cause's proof.
Files
Updated by Sage Weil almost 7 years ago
- Status changed from New to 12
- Priority changed from Normal to Immediate
Updated by Sage Weil almost 7 years ago
Igor, as far as I can tell _flush_bdev_safely is already being called from _sync_and_flush_log. Can you explain the steps you're doing to reproduce this?
Thanks!
Updated by Igor Fedotov almost 7 years ago
That was a standalone bluestore executed as FIO plugin. And random 4K overwrite scenario - 128 Gb total, 32K objects. I'm not completely sure that bluefs file in question is log_writer - that was just an assumption.
Updated by Igor Fedotov almost 7 years ago
- File aLL4.fio aLL4.fio added
- File ceph-5.conf ceph-5.conf added
- File ceph-bluestore-somnath.conf ceph-bluestore-somnath.conf added
- File w4-5.fio w4-5.fio added
FIO jobs and configs to precondition the store and execute the overwrite. Paths to be updated!
To run:
fio aLL4.fio
fio w4-5.fio
Please note FIO RSS memory for the second command.I saw >7 Gb by the end of the process
Updated by Igor Fedotov over 6 years ago
When using Jemalloc still can observe some excessive memory usage (approx. x2 increase over configured cache limit) for random reads FIO/Bluestore scenario. BlueStore buffered read (and hence object content caching) is on.
Not reproducible for TCMalloc.
Suggest to close as JEMalloc isn't considered production-ready for Ceph at the moment.
Updated by Sage Weil over 6 years ago
- Subject changed from Bluestore+BlueFS causes unexpected RSS Memory usage growth to jemalloc+Bluestore+BlueFS causes unexpected RSS Memory usage growth
- Priority changed from High to Low
Updated by Sage Weil about 6 years ago
- Status changed from 12 to Won't Fix
we don't care about jemalloc at this point