https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2018-01-09T01:05:19ZCeph bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1045512018-01-09T01:05:19ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>bluestore</i></li></ul> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1046432018-01-10T03:03:19Zfrank lin
<ul></ul><p>The work load of read throughput test is 6 fio server with the following parameter</p>
<p>[4m-seq]<br />description="4m-seq-read" <br />direct=1<br />ioengine=libaio<br />directory=/mnt/cephfs/fio_benchmark/4m/<br />numjobs=24<br />iodepth=4<br />group_reporting<br />rw=read<br />bs=4M<br />size=5G</p>
<p>my osd node has only about 1GB ram for 1 osd.So if bluestore_cache_data uses too much memory then the osd got killed by oom killer.</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1046592018-01-10T14:53:26ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul><p>Writes that are in flight to disk show up under bluestore_cache_data, so even if it is not <strong>caching</strong> anything you'll still see that value be non-zero. This should only happen while under load, however.</p>
<p>What is your memory limit set to? If your memory limits are that tight you probably need to adjust osd_client_message_size_cap, the total amount of memory we are allowed to consume with incoming in-flight requests (default is 500 MB!).</p>
<p>As far as bluestore_cache_data is concerned, I think there is only a bug if the value stays non-zero when the osd goes idle...</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1046732018-01-10T16:02:41Zfrank lin
<ul></ul><p>Sage Weil wrote:</p>
<blockquote>
<p>Writes that are in flight to disk show up under bluestore_cache_data, so even if it is not <strong>caching</strong> anything you'll still see that value be non-zero. This should only happen while under load, however.</p>
<p>What is your memory limit set to? If your memory limits are that tight you probably need to adjust osd_client_message_size_cap, the total amount of memory we are allowed to consume with incoming in-flight requests (default is 500 MB!).</p>
<p>As far as bluestore_cache_data is concerned, I think there is only a bug if the value stays non-zero when the osd goes idle...</p>
</blockquote>
<p>About "Writes that are in flight to disk show up under bluestore_cache_data", when bluestore_cache_data took more than 700MB ram, I was running 100% read throughput test with zero write.</p>
<p>The total read throughput was about 2.4GB/s for my 48 osds cluster, but some of the osd would get oom killed in minutes if I ran my cluster at that throughput.</p>
<p>When I ran write throughput test the bluestore_cache_data took only 0-20MB ram.</p>
<p>I also found bluestore_cache_data will take more ram as the reading throughput increased if I keep the read throughput capped the situation would be much better.</p>
<p>PS:</p>
<p>I did set osd_client_message_size_cap to 64MB(but I haven't found any detailed information about this setting.Is it a cache for incoming data? Will it accounted in mempool?And I haven't notice set osd_client_message_size_cap to 64MB reduced memory usage)</p>
<p>My memory limit is 1GB for one osd, I know it is very small but I am running ceph on arm server with only 2GB memory.</p>
<p>It will be highly appreciated if you have any advice to reduce the memory usage.</p>
<p>Thanks</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1047052018-01-11T07:35:57Zfrank lin
<ul></ul><p>One more fact of my test to add.<br />I have 48 osd for the test and there were only a few of the osd's bluestore_cache_data went up to about 200-700MB.The rest of the osds are at about 10-30MB.</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1053722018-01-19T15:03:47ZSage Weilsage@newdream.net
<ul></ul><p>Two things to try:</p>
<p>- bluestore_default_buffered_read = false should make the problem go away, but is more of a workaround</p>
<p>- bluestore_cache_trim_interval = .05 should reduce the usage (dfeault is .2 = trim every 200ms).. i think this is the actual problem!</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1053762018-01-19T15:28:58Zfrank lin
<ul></ul><p>Sage Weil wrote:</p>
<blockquote>
<p>Two things to try:</p>
<p>- bluestore_default_buffered_read = false should make the problem go away, but is more of a workaround</p>
<p>- bluestore_cache_trim_interval = .05 should reduce the usage (dfeault is .2 = trim every 200ms).. i think this is the actual problem!</p>
</blockquote>
<p>Thanks for the advice.<br />I have tried to set bluestore_cache_trim_interval to 0.05 and it did lower the memory use but if the read load is high enough some osds would still use more than 500mb in bluestore_cache_data.</p>
<p>Is there a explain why only some osd uses much more memory than others?</p>
<p>I will try bluestore_default_buffered_read = false</p>
<p>BTW:<br />Is there any advice to further reduce memory usage to make 1GB one osd work without oom problem?</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1054392018-01-22T06:00:00Zfrank lin
<ul></ul><p>I did some test with bluestore_default_buffered_read = false</p>
<p>The bluestore_cache_data now only use around a few KB of memory now, but buffer_anon uses around 500MB memory now(this happens only on some of the osds,the osds use more buffer_anon are the same osds that use more bluestore_cache_data).</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1071312018-02-08T15:50:32ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>12</i></li></ul><p>Ok, I think the thing to do here is make the bluestore trimming a bit more frequent, and have this as a known caveat for very low-memory situations.</p>
<p>Or, perhaps, we could make the read buffered option change if the cache size is below some threshold...</p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1077022018-02-20T14:35:58ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/20498">https://github.com/ceph/ceph/pull/20498</a></p> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1082622018-02-28T14:00:39ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li><li><strong>Backport</strong> set to <i>luminous</i></li></ul> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1084952018-03-05T17:03:13ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/23226">Backport #23226</a>: luminous: bluestore_cache_data uses too much memory</i> added</li></ul> bluestore - Bug #22616: bluestore_cache_data uses too much memoryhttps://tracker.ceph.com/issues/22616?journal_id=1104052018-04-06T20:39:10ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>