Infrastructure

http://pulpito.front.sepia.ceph.com/yuriw-2022-05-04_20:09:21-rados-pacific-distro-default-smithi/

Octopus too:

/a/yuriw-2022-04-26_20:58:55-rados-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6807537

SELinux denials found on ubuntu@smithi186.front.sepia.ceph.com: ['type=AVC msg=audit(1651114126.589:7615): avc: denied { ioctl } for pid=42421 comm="iptables" path="/var/lib/containers/storage/overlay/45b930a7fa22d5f3f9b2b1b82e1fa0c602112ec6d6aa1e8625f5bc8c8f39a8e0/merged" dev="overlay" ino=3541410 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1', 'type=AVC msg=audit(1651114126.773:7617): avc: denied { ioctl } for pid=42454 comm="iptables" path="/var/lib/containers/storage/overlay/45b930a7fa22d5f3f9b2b1b82e1fa0c602112ec6d6aa1e8625f5bc8c8f39a8e0/merged" dev="overlay" ino=3541410 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1'] 

Looks like this is extremely similar to https://bugzilla.redhat.com/show_bug.cgi?id=2031022 and is likely the exact same issue in centos.

Some occurrences here: https://pulpito.ceph.com/yuriw-2022-07-19_23:25:12-rados-wip-yuri2-testing-2022-07-15-0755-pacific-distro-default-smithi/

/a/yuriw-2022-11-23_21:33:04-rados-wip-yuri4-testing-2022-11-10-1051-distro-default-smithi/7089721

/a/lflores-2023-02-08_20:25:06-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/7161983

/a/yuriw-2023-02-24_17:50:19-rados-main-distro-default-smithi/7186690

/a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7192742

/a/yuriw-2023-03-30_21:53:20-rados-wip-yuri7-testing-2023-03-29-1100-distro-default-smithi/7228104

/a/yuriw-2023-04-04_15:24:40-rados-wip-yuri4-testing-2023-03-31-1237-distro-default-smithi/7231172

/a/yuriw-2023-04-04_21:18:37-rados-wip-yuri3-testing-2023-04-04-0833-pacific-distro-default-smithi/7231994

/a/yuriw-2023-04-25_14:15:40-rados-pacific-release-distro-default-smithi/7251604

/a/yuriw-2023-04-25_18:56:08-rados-wip-yuri5-testing-2023-04-25-0837-pacific-distro-default-smithi/7252542

/a/yuriw-2023-04-24_22:54:45-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7250642

/a/yuriw-2023-04-26_01:16:19-rados-wip-yuri11-testing-2023-04-25-1605-pacific-distro-default-smithi/7254115

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276289

Laura Flores wrote:

Hey Brad, any idea on how we can silence these selinux denials?

Not yet, looks like another system issue rather than a ceph problem. I'll try and reproduce this and analyse it in more depth, I can then maybe make a recommendation.

Thank you Brad!

Reproduced this in interactive mode.

# ausearch -i -m avc,user_avc --comm iptables
----
type=PROCTITLE msg=audit(05/18/2023 23:55:08.726:20155) : proctitle=/usr/sbin/iptables --version
type=EXECVE msg=audit(05/18/2023 23:55:08.726:20155) : argc=2 a0=/usr/sbin/iptables a1=--version
type=SYSCALL msg=audit(05/18/2023 23:55:08.726:20155) : arch=x86_64 syscall=execve success=yes exit=0 a0=0xc0000aa300 a1=0xc00009c0d8 a2=0xc0000b6380 a3=0x8 items=0 ppid=110175 pid=110229 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=iptables exe=/usr/sbin/xtables-nft-multi subj=system_u:system_r:iptables_t:s0 key=(null)                                                                              
type=AVC msg=audit(05/18/2023 23:55:08.726:20155) : avc:  denied  { ioctl } for  pid=110229 comm=iptables path=/var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged dev="overlay" ino=3281139 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1

# journalctl -t setroubleshoot |grep "ioctl" 
May 18 23:55:11 smithi062 setroubleshoot[110245]: SELinux is preventing /usr/sbin/xtables-nft-multi from ioctl access on the directory /var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged. For complete SELinux messages run: sealert -l 1c6eb49b-27ef-4a02-afc6-3709c6d6f60e
May 18 23:55:11 smithi062 setroubleshoot[110245]: SELinux is preventing /usr/sbin/xtables-nft-multi from ioctl access on the directory /var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged.
                                                  If you believe that xtables-nft-multi should be allowed ioctl access on the merged directory by default.

# sealert -l 1c6eb49b-27ef-4a02-afc6-3709c6d6f60e
SELinux is preventing /usr/sbin/xtables-nft-multi from ioctl access on the directory /var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that xtables-nft-multi should be allowed ioctl access on the merged directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'iptables' --raw | audit2allow -M my-iptables
# semodule -X 300 -i my-iptables.pp

Additional Information:
Source Context                system_u:system_r:iptables_t:s0
Target Context                system_u:object_r:container_file_t:s0:c1022,c1023
Target Objects                /var/lib/containers/storage/overlay/003778e0b0ce4b
                              df70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01
                              /merged [ dir ]
Source                        iptables
Source Path                   /usr/sbin/xtables-nft-multi
Port                          <Unknown>
Host                          smithi062
Source RPM Packages           iptables-1.8.4-22.el8.x86_64
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Local Policy RPM              selinux-policy-targeted-3.14.3-95.el8_6.4.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Permissive
Host Name                     smithi062
Platform                      Linux smithi062 4.18.0-372.9.1.el8.x86_64 #1 SMP
                              Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64
Alert Count                   1
First Seen                    2023-05-18 23:55:08 UTC
Last Seen                     2023-05-18 23:55:08 UTC
Local ID                      1c6eb49b-27ef-4a02-afc6-3709c6d6f60e

Raw Audit Messages
type=AVC msg=audit(1684454108.726:20155): avc:  denied  { ioctl } for  pid=110229 comm="iptables" path="/var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged" dev="overlay" ino=3281139 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1

type=SYSCALL msg=audit(1684454108.726:20155): arch=x86_64 syscall=execve success=yes exit=0 a0=c0000aa300 a1=c00009c0d8 a2=c0000b6380 a3=8 items=0 ppid=110175 pid=110229 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=iptables exe=/usr/sbin/xtables-nft-multi subj=system_u:system_r:iptables_t:s0 key=(null)

Hash: iptables,iptables_t,container_file_t,dir,ioctl

This is well beyond my understanding of iptables and selinux but it appears that we may be running iptables from a container and that selinux doesn't like that?

Looks very similar, but not identical to, https://bugzilla.redhat.com/show_bug.cgi?id=2134820 which makes me think it is yet another system issue (not a ceph problem). Traditionally the solution to these has been to add a generic regex to the known_denials list in teuthology/task/selinux.py but recently there has been justified resistence to this since there are so many and a lot of them are very generic. The risk of filtering out legitimate future issues is substantial. Even on this single, current run we actually see all of these denials, but only pick up the iptables one.

# journalctl -t setroubleshoot |grep "SELinux is preventing" 
May 18 23:31:28 smithi062.front.sepia.ceph.com setroubleshoot[1632]: SELinux is preventing /usr/bin/ping from node_bind access on the icmp_socket labeled node_t. For complete SELinux messages run: sealert -l dfad7777-89d9-47ef-b9e4-5bff60e1c42a
May 18 23:31:28 smithi062.front.sepia.ceph.com setroubleshoot[1632]: SELinux is preventing /usr/bin/ping from node_bind access on the icmp_socket labeled node_t.
May 18 23:32:07 smithi062 setroubleshoot[2412]: SELinux is preventing /usr/sbin/rsyslogd from search access on the directory /home/ubuntu/cephtest/archive/syslog/misc.log. For complete SELinux messages run: sealert -l 1ea476a7-e579-463b-903c-58e097492c9d
May 18 23:32:07 smithi062 setroubleshoot[2412]: SELinux is preventing /usr/sbin/rsyslogd from search access on the directory /home/ubuntu/cephtest/archive/syslog/misc.log.
May 18 23:38:32 smithi062 setroubleshoot[71423]: SELinux is preventing /usr/sbin/rsyslogd from search access on the directory /home/ubuntu/cephtest/archive/syslog/kern.log. For complete SELinux messages run: sealert -l 1ea476a7-e579-463b-903c-58e097492c9d
May 18 23:38:32 smithi062 setroubleshoot[71423]: SELinux is preventing /usr/sbin/rsyslogd from search access on the directory /home/ubuntu/cephtest/archive/syslog/kern.log.
May 18 23:40:14 smithi062 setroubleshoot[81928]: SELinux is preventing /usr/sbin/sssd from read access on the file resolv.conf. For complete SELinux messages run: sealert -l 193ffed4-49ec-4845-94aa-84bc8c6d1d0e
May 18 23:40:14 smithi062 setroubleshoot[81928]: SELinux is preventing /usr/sbin/sssd from read access on the file resolv.conf.
May 18 23:55:11 smithi062 setroubleshoot[110245]: SELinux is preventing /usr/sbin/xtables-nft-multi from ioctl access on the directory /var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged. For complete SELinux messages run: sealert -l 1c6eb49b-27ef-4a02-afc6-3709c6d6f60e
May 18 23:55:11 smithi062 setroubleshoot[110245]: SELinux is preventing /usr/sbin/xtables-nft-multi from ioctl access on the directory /var/lib/containers/storage/overlay/003778e0b0ce4bdf70dac1df6825862a4bd16c3ea71ca6b49bfbe8105adacd01/merged.
May 19 00:03:38 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from 'read, write' accesses on the file /var/lib/rpm/.dbenv.lock. For complete SELinux messages run: sealert -l 1fd5423a-95a2-4efa-8c5f-8535fd97bf11
May 19 00:03:38 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from 'read, write' accesses on the file /var/lib/rpm/.dbenv.lock.
May 19 00:03:40 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from 'read, write' accesses on the file /var/lib/rpm/.dbenv.lock. For complete SELinux messages run: sealert -l 1fd5423a-95a2-4efa-8c5f-8535fd97bf11
May 19 00:03:40 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from 'read, write' accesses on the file /var/lib/rpm/.dbenv.lock.
May 19 00:03:42 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from lock access on the file /var/lib/rpm/.dbenv.lock. For complete SELinux messages run: sealert -l f34f8acc-ad05-4da6-9ab2-c131d33513d5
May 19 00:03:42 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from lock access on the file /var/lib/rpm/.dbenv.lock.
May 19 00:03:45 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from getattr access on the file /var/lib/rpm/__db.001. For complete SELinux messages run: sealert -l 5d20203f-e24f-44f2-a80f-65c49eaa3482
May 19 00:03:45 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from getattr access on the file /var/lib/rpm/__db.001.
May 19 00:03:47 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from map access on the file /var/lib/rpm/__db.001. For complete SELinux messages run: sealert -l daeb1068-a144-4b69-a85f-ce291b413bb7
May 19 00:03:47 smithi062 setroubleshoot[115691]: SELinux is preventing /usr/libexec/platform-python3.6 from map access on the file /var/lib/rpm/__db.001.

The fact we just seem to allow and ever increasing range of these and that they never seem to get resolved is concerning.

/a/yuriw-2023-05-28_14:41:12-rados-reef-release-distro-default-smithi/7288799

/a/yuriw-2023-05-31_21:58:42-rados-wip-yuri3-testing-2023-05-31-0931-reef-distro-default-smithi/7293085

/a/yuriw-2023-06-13_18:33:48-rados-wip-yuri10-testing-2023-06-02-1406-distro-default-smithi/7302915

/a/yuriw-2023-06-22_20:29:56-rados-wip-yuri3-testing-2023-06-22-0812-reef-distro-default-smithi/7313203

/a/yuriw-2023-07-10_18:41:02-rados-wip-yuri6-testing-2023-07-10-0816-distro-default-smithi/7332613/

/a/yuriw-2023-07-18_14:02:17-rados-wip-yuri6-testing-2023-07-17-0838-reef-distro-default-smithi/7342958

/a/yuriw-2023-07-20_16:07:02-rados-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7345233/

/a/yuriw-2023-08-22_18:16:03-rados-wip-yuri10-testing-2023-08-17-1444-distro-default-smithi/7376699

/a/yuriw-2023-08-15_18:58:56-rados-wip-yuri3-testing-2023-08-15-0955-distro-default-smithi/7369450

Hey @Brad, any tips for resolving this? Would you recommend whitelisting, or should we consider a different approach?

Hey @Laura,

See https://tracker.ceph.com/issues/55347#note-23 The short answer is, I'm not sure what the alternative to whitelisting is but my last attempt to whitelist one of these was rejected. This is really not a ceph issue AFAICT other than it's stopping the tests from completing.