https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2019-07-31T21:07:25ZCeph bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422242019-07-31T21:07:25ZNeha Ojhanojha@redhat.com
<ul></ul><p>Which version of Ceph are you running?</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422252019-07-31T21:25:23ZNeha Ojhanojha@redhat.com
<ul></ul><p>Can you enable debug_osd=10 and see what this line (<a class="external" href="https://github.com/ceph/ceph/commit/fc3bdad87597066a813a3734b2a79e803340be36#diff-a9faffcf40600fd57aea5451cef5abe9R4214">https://github.com/ceph/ceph/commit/fc3bdad87597066a813a3734b2a79e803340be36#diff-a9faffcf40600fd57aea5451cef5abe9R4214</a>) reports in the log?</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422682019-08-01T17:38:50ZBen Englandbengland@redhat.com
<ul><li><strong>Target version</strong> set to <i>v14.2.3</i></li></ul><p>version you asked for:</p>
<p>ceph-base-14.2.2-0.el7.x86_64</p>
<p>from the Ceph container image ceph/ceph:v14.2.2-20190722</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422692019-08-01T17:44:07ZBen Englandbengland@redhat.com
<ul></ul>from Joe T on his system:
<ol>
<li>ceph version<br />ceph version 14.2.2-218-g734b519 (734b5199dc45d3d36c8d8d066d6249cc304d0e0e) nautilus (stable)</li>
</ol> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422732019-08-01T19:17:39ZNeha Ojhanojha@redhat.com
<ul></ul><p>Neha Ojha wrote:</p>
<blockquote>
<p>Can you enable debug_osd=10 and see what this line (<a class="external" href="https://github.com/ceph/ceph/commit/fc3bdad87597066a813a3734b2a79e803340be36#diff-a9faffcf40600fd57aea5451cef5abe9R4214">https://github.com/ceph/ceph/commit/fc3bdad87597066a813a3734b2a79e803340be36#diff-a9faffcf40600fd57aea5451cef5abe9R4214</a>) reports in the log?</p>
</blockquote>
<p>my bad, it should be debug_bluestore=10</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1422762019-08-01T20:36:09ZJosh Durgin
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p>Joe Talerico reproduced this and found the POD_LIMIT was getting set, but not the system-wide limit, so the current OSD code reading only the system limit would have no effect.</p>
<p>These info available in the osd container was:<br /><pre>
cat /proc/80946/cgroup
12:pids:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
11:perf_event:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
10:devices:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
9:hugetlb:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
8:freezer:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
7:blkio:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
6:cpu,cpuacct:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
5:memory:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
4:rdma:/
3:net_cls,net_prio:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
2:cpuset:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
1:name=systemd:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/crio-7d8c95780ecfe701043003a914b4ab2cb410c0139478e46db8f07008ba8e733e.scope
cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688470ce_b491_11e9_b8b1_98039b616b98.slice/memory.limit_in_bytes
6442450944
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
</pre></p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1423752019-08-05T14:09:27ZBen Englandbengland@redhat.com
<ul></ul><p>My guess would be if CGroup limit is X, then 0.95 X - 1/2 GB should be fine for osd_memory_target, that would give the OSDs time to detect that they are over their limit and purge the cache to bring it back down, while wasting as little memory resource as possible.</p>
<p>The worst case here is really intense I/O to NVM devices, where the cache size can increase rapidly before the OSD can detect that there is a problem. In the past, HDDs limited the IOPS rate.</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1423892019-08-05T16:05:27ZMark Nelsonmark.a.nelson@gmail.com
<ul></ul><p>@ben that's probably a semi-reasonable assumption in a lot of cases, though I've noticed that the kernel doesn't always reclaim unmapped memory right away which can make this really tricky. CentOS and RHEL seem to do better than Ubuntu does though and I have no idea how containers interact with it.</p>
<p>Typically I just say to give them an extra 20% but I could be convinced something like 5%+XMB could also work.</p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1425362019-08-06T16:16:02ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Backport</strong> set to <i>nautilus</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/29511">https://github.com/ceph/ceph/pull/29511</a></p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1426782019-08-08T20:52:24ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul><p>nautilus backport: <a class="external" href="https://github.com/ceph/ceph/pull/29562">https://github.com/ceph/ceph/pull/29562</a></p> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1430752019-08-15T09:08:13ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-5 priority-high3 closed" href="/issues/41273">Backport #41273</a>: nautilus: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratio</i> added</li></ul> bluestore - Bug #41037: Containerized cluster failure due to osd_memory_target not being set to ratio of cgroup_limit per osd_memory_target_cgroup_limit_ratiohttps://tracker.ceph.com/issues/41037?journal_id=1445932019-08-27T14:54:35ZJosh Durgin
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>