Bug #55443
open"SELinux denials found.." in rados run
Added by Yuri Weinstein about 2 years ago. Updated 6 months ago.
0%
Description
Run: http://pulpito.front.sepia.ceph.com/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/
Jobs: ['6803080', '6803087', '6803132', '6803104', '6803113', '6803128', '6803112', '6803134', '6803107', '6803103']
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/6803080/teuthology.log
SELinux denials found on ubuntu@smithi120.front.sepia.ceph.com: ['type=AVC msg=audit(1650731309.430:200): avc: denied { node_bind } for pid=1997 comm="ping" saddr=172.21.15.120 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
Updated by Brad Hubbard about 2 years ago
This looks like another systems issue, similar in nature to https://bugzilla.redhat.com/show_bug.cgi?id=1848929 so I think the only way we can get the tests passing until this is resolved in centos is to mask these selinux denials in the teuthology code so they are ignored.
Going to move this to infrastructure so David can give his input.
Updated by Brad Hubbard about 2 years ago
- Project changed from Ceph to Infrastructure
Updated by David Galloway about 2 years ago
Yeah, I don't see anything indicating Ceph is doing the ping so it should be fine to whitelist.
Updated by Yuri Weinstein about 2 years ago
This is for 16.2.8
Updated by Yuri Weinstein about 2 years ago
- Priority changed from Normal to Urgent
Updated by Laura Flores almost 2 years ago
Octopus too:
/a/yuriw-2022-04-26_20:58:55-rados-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6807537
SELinux denials found on ubuntu@smithi186.front.sepia.ceph.com: ['type=AVC msg=audit(1651114126.589:7615): avc: denied { ioctl } for pid=42421 comm="iptables" path="/var/lib/containers/storage/overlay/45b930a7fa22d5f3f9b2b1b82e1fa0c602112ec6d6aa1e8625f5bc8c8f39a8e0/merged" dev="overlay" ino=3541410 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1651114126.773:7617): avc: denied { ioctl } for pid=42454 comm="iptables" path="/var/lib/containers/storage/overlay/45b930a7fa22d5f3f9b2b1b82e1fa0c602112ec6d6aa1e8625f5bc8c8f39a8e0/merged" dev="overlay" ino=3541410 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
Updated by Brad Hubbard almost 2 years ago
Laura Flores wrote:
Octopus too:
/a/yuriw-2022-04-26_20:58:55-rados-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6807537
[...]
@Laura I think we are looking at that one in https://tracker.ceph.com/issues/55347 right?
Updated by Laura Flores almost 2 years ago
Brad Hubbard wrote:
@Laura I think we are looking at that one in https://tracker.ceph.com/issues/55347 right?
Ah yes, thanks Brad.
Updated by Sridhar Seshasayee almost 2 years ago
/a/yuriw-2022-06-15_18:29:33-rados-wip-yuri4-testing-2022-06-15-1000-pacific-distro-default-smithi/6881417
Updated by Kamoltat (Junior) Sirivadhna over 1 year ago
/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958123
Updated by Kamoltat (Junior) Sirivadhna over 1 year ago
/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958226
Updated by Kamoltat (Junior) Sirivadhna over 1 year ago
/a/yuriw-2022-08-04_11:58:29-rados-wip-yuri3-testing-2022-08-03-0828-pacific-distro-default-smithi/6958331
Updated by Laura Flores over 1 year ago
/a/yuriw-2022-08-11_16:46:00-rados-wip-yuri3-testing-2022-08-11-0809-pacific-distro-default-smithi/6968124
Updated by Matan Breizman over 1 year ago
/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986470
/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986482
/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986492
Updated by Nitzan Mordechai over 1 year ago
/a/yuriw-2023-01-21_17:58:46-rados-wip-yuri6-testing-2023-01-20-0728-distro-default-smithi/7132689
Updated by Nitzan Mordechai over 1 year ago
/a/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137279
/a/yuriw-2023-01-24_22:27:30-rados-wip-yuri6-testing-2023-01-24-0746-distro-default-smithi/7137405
Updated by Brad Hubbard about 1 year ago
Reproduced this in interactive mode.
# ausearch -i -m avc,user_avc --comm ping
----
type=PROCTITLE msg=audit(12/18/2022 19:21:30.794:51) : proctitle=ping -I 172.21.15.22 -nq -c1 172.21.0.1
type=SYSCALL msg=audit(12/18/2022 19:21:30.794:51) : arch=x86_64 syscall=bind success=yes exit=0 a0=0x3 a1=0x5612c819e070 a2=0x10 a3=0x7f7599452480 items=0 ppid=5925 pid=5926 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ping exe=/usr/bin/ping subj=system_u:system_r:ping_t:s0 key=(null)
type=AVC msg=audit(12/18/2022 19:21:30.794:51) : avc: denied { node_bind } for pid=5926 comm=ping saddr=172.21.15.22 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1
----
type=PROCTITLE msg=audit(02/22/2023 03:55:28.252:48) : proctitle=ping -I 172.21.15.149 -nq -c1 172.21.0.1
type=SYSCALL msg=audit(02/22/2023 03:55:28.252:48) : arch=x86_64 syscall=bind success=yes exit=0 a0=0x3 a1=0x55ddd3910070 a2=0x10 a3=0x7f777ab1d480 items=0 ppid=1618 pid=1619 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ping exe=/usr/bin/ping subj=system_u:system_r:ping_t:s0 key=(null)
type=AVC msg=audit(02/22/2023 03:55:28.252:48) : avc: denied { node_bind } for pid=1619 comm=ping saddr=172.21.15.149 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1
----
type=PROCTITLE msg=audit(02/22/2023 04:10:06.100:47) : proctitle=ping -I 172.21.15.149 -nq -c1 172.21.0.1
type=SYSCALL msg=audit(02/22/2023 04:10:06.100:47) : arch=x86_64 syscall=bind success=yes exit=0 a0=0x3 a1=0x5591f49ce070 a2=0x10 a3=0x7f5db4899480 items=0 ppid=3175 pid=3176 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=ping exe=/usr/bin/ping subj=system_u:system_r:ping_t:s0 key=(null)
type=AVC msg=audit(02/22/2023 04:10:06.100:47) : avc: denied { node_bind } for pid=3176 comm=ping saddr=172.21.15.149 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1
# journalctl -t setroubleshoot |grep "ping.*sealert" Feb 22 04:10:12 smithi149.front.sepia.ceph.com setroubleshoot[3181]: SELinux is preventing /usr/bin/ping from node_bind access on the icmp_socket labeled node_t. For complete SELinux messages run: sealert -l 1d3b7b2c-5383-447d-81a8-95126a16cf1e
# sealert -l 1d3b7b2c-5383-447d-81a8-95126a16cf1e
SELinux is preventing /usr/bin/ping from node_bind access on the icmp_socket labeled node_t.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that ping should be allowed node_bind access on icmp_socket labeled node_t by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'ping' --raw | audit2allow -M my-ping
# semodule -X 300 -i my-ping.pp
Additional Information:
Source Context system_u:system_r:ping_t:s0
Target Context system_u:object_r:node_t:s0
Target Objects Unknown [ icmp_socket ]
Source ping
Source Path /usr/bin/ping
Port <Unknown>
Host smithi149.front.sepia.ceph.com
Source RPM Packages iputils-20180629-9.el8.x86_64
Target RPM Packages
SELinux Policy RPM selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Local Policy RPM selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name smithi149.front.sepia.ceph.com
Platform Linux smithi149.front.sepia.ceph.com
4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15
22:12:19 EDT 2022 x86_64 x86_64
Alert Count 2
First Seen 2023-02-22 03:55:28 UTC
Last Seen 2023-02-22 04:10:06 UTC
Local ID 1d3b7b2c-5383-447d-81a8-95126a16cf1e
Raw Audit Messages
type=AVC msg=audit(1677039006.100:47): avc: denied { node_bind } for pid=3176 comm="ping" saddr=172.21.15.149 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1
type=SYSCALL msg=audit(1677039006.100:47): arch=x86_64 syscall=bind success=yes exit=0 a0=3 a1=5591f49ce070 a2=10 a3=7f5db4899480 items=0 ppid=3175 pid=3176 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=ping exe=/usr/bin/ping subj=system_u:system_r:ping_t:s0 key=(null)
Hash: ping,ping_t,node_t,icmp_socket,node_bind
This is intermittent. It doesn't happen every time a particular test is run, but it happens a lot so a lot of tests fail. It's also not readily reproducible on a system where the error has been seen. I can run 'ping -I 172.21.15.149 -nq -c1 172.21.0.1' as various users on this system and not reproduce the errors.
There is enough evidence to suggest that this is a system issue (https://bugzilla.redhat.com/show_bug.cgi?id=1803759) so I am going to add it to the blocklist to get this resolved finally.
Updated by Laura Flores 12 months ago
Hey Brad, were you working on a PR for this? If so, can you link it?
Is this it? https://github.com/ceph/teuthology/pull/1830
Updated by Brad Hubbard 12 months ago
Laura Flores wrote:
Hey Brad, were you working on a PR for this? If so, can you link it?
Is this it? https://github.com/ceph/teuthology/pull/1830
Yes.
Updated by Aishwarya Mathuria 9 months ago
Saw this in a Pacific run - /a/yuriw-2023-08-16_22:40:18-rados-wip-yuri2-testing-2023-08-16-1142-pacific-distro-default-smithi/7371220
Updated by Laura Flores 6 months ago
- Translation missing: en.field_tag_list set to test-failure
Seen in a rados/standalone job, which seems new:
Description: rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd-backfill}
/a/yuriw-2023-11-10_18:18:41-rados-wip-yuri3-testing-2023-11-09-1355-quincy-distro-default-smithi/7454517
2023-11-10T23:08:34.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0
2023-11-10T23:08:34.567 INFO:teuthology.orchestra.run.smithi042.stdout:type=AVC msg=audit(1699647988.925:205): avc: denied { node_bind } for pid=1942 comm="ping" saddr=172.21.15.42 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1
2023-11-10T23:08:34.568 DEBUG:teuthology.task.selinux:ubuntu@smithi042.front.sepia.ceph.com has 1 denials
2023-11-10T23:08:34.568 ERROR:teuthology.run_tasks:Manager failed: selinux
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/__init__.py", line 136, in __exit__
self.teardown()
File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/selinux.py", line 166, in teardown
self.get_new_denials()
File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/selinux.py", line 215, in get_new_denials
raise SELinuxError(node=remote,
teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi042.front.sepia.ceph.com: ['type=AVC msg=audit(1699647988.925:205): avc: denied { node_bind } for pid=1942 comm="ping" saddr=172.21.15.42 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
Updated by Laura Flores 6 months ago
- Has duplicate Bug #63518: Selinux denial in rados/standalone job added