https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2014-10-29T03:46:17ZCeph Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=437972014-10-29T03:46:17ZDan van der Ster
<ul></ul><p>It is triggered by fixfiles -C /etc/selinux/targeted/contexts/files/file_contexts.pre restore</p>
<pre>
| `-yum,47342 /usr/bin/yum --skip-broken -x ceph* -x libceph* -x librados* -x librbd* -x kernel* -x ...
| `-sh,51420 /var/tmp/rpm-tmp.GxmtU1 2
| `-fixfiles,51822 /sbin/fixfiles -C /etc/selinux/targeted/contexts/files/file_contexts.pre restore
| `-restorecon,51873 -i -f - -R -p -e /sys -e /proc -e /dev -e /mnt -e /var/tmp -e /home -e /tmp -e ...
</pre> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=437982014-10-29T05:25:18ZDan van der Ster
<ul></ul><p>Here's a solution:</p>
<pre>
echo "/var/lib/ceph/" >> /etc/selinux/fixfiles_exclude_dirs
</pre> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=440802014-11-05T20:30:41ZWade Mealingwmealing@redhat.com
<ul></ul><p>I would strongly reccomend limiting it to the subdirectories where large mounts are, not on the parent directory. This would probably solve the issue of having the relabeling eating into IO performance of those disks.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526462015-05-28T16:08:30ZKen Dreyerkdreyer@redhat.com
<ul><li><strong>Regression</strong> set to <i>No</i></li></ul><p>Milan, how can we implement a fix for this so it works out-of-the-box? Is selinux-policy-targeted the right place to fix this?</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526642015-05-28T17:48:13ZMilan Brozmbroz@redhat.com
<ul></ul><p>That "fix" can make things worse later... It is probably good for quick workaround for some particular case though.<br />IMHO the proper fix is to apply new selinux policy for ceph, we should start to test what we already have and will see if it is doable in some shorter term.</p>
<p>Anyway, it would be good to have this tracked in bugzilla for RHEL - that way RHEL selinux-policy maintainer can comment it.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526662015-05-28T18:38:34ZBoris Rantobranto@redhat.com
<ul></ul><p>This is (well, will be) an intended behaviour, soon. We need to relabel the files for the SELinux policy to take effect (once it will be available).</p>
<p>That being said, the fixfiles script could probably be improved for better performance -- i.e. updating files based on policy changes or running single thread per single hdd/mount point to improve performance of the call.</p>
<p>btw: Do you see this behaviour also on rhel 7 and current range of fedoras? There might have been some performance improvements to this behaviour in later releases of selinux tools and these could technically get backported although it might already be too late for such a big change to rhel 6 environment in the rhel 6 release cycle.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526672015-05-28T19:33:48ZKen Dreyerkdreyer@redhat.com
<ul></ul><p>Milan, one of the things Boris and I discussed is that Dan's strace shows a lot of <code>stat()</code> calls there. So even if the Ceph policy itself isn't changing, it still takes a while for <code>restorecon</code> to <code>stat()</code> everything under <code>/var/lib/ceph/osd</code>. I'm not sure what "a while" is, though (seconds, minutes, etc)?</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526772015-05-29T07:04:40ZDan van der Ster
<ul></ul><p>Ken Dreyer wrote:</p>
<blockquote>
<p>I'm not sure what "a while" is, though (seconds, minutes, etc)?</p>
</blockquote>
<p>It depends on the number of objects, of course. In our case it was taking 10's of minutes, hours in some cases. This was pretty nasty since the drives were pinned to 100% in iostat throughout the running of fixfiles and users were definitely suffering.</p>
<p>If this is a necessary operation, in the least fixfiles should run with a lower ionice priority, (e.g. -c2 -n7 like the mlocate daily cron), or best, ionice'd to idle.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=526782015-05-29T07:06:05ZDan van der Ster
<ul></ul><p>Boris Ranto wrote:</p>
<blockquote>
<p>btw: Do you see this behaviour also on rhel 7 and current range of fedoras?</p>
</blockquote>
<p>I don't have a rhel7 ceph-osd server yet, so I can't comment.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=890392017-04-12T16:37:28ZGreg Farnumgfarnum@redhat.com
<ul></ul><p>SELinux is used against Ceph now.</p> Ceph - Bug #9927: RHEL: selinux-policy-targeted rpm update triggers slow requests https://tracker.ceph.com/issues/9927?journal_id=890402017-04-12T16:37:39ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Can't reproduce</i></li></ul><p>pls reopen if this is a problem on rhel7</p>