Bug #15524: "SELinux denials" in ceph-deploy-wip-sage-testing2-distro-basic-mira - Ceph - Ceph

Actions

Copy link

Bug #15524

closed

"SELinux denials" in ceph-deploy-wip-sage-testing2-distro-basic-mira

Added by Yuri Weinstein about 8 years ago. Updated almost 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

ceph-deploy

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2016-04-15_14:27:43-ceph-deploy-wip-sage-testing2-distro-basic-mira/
Jobs: all centos
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-04-15_14:27:43-ceph-deploy-wip-sage-testing2-distro-basic-mira/132734/teuthology.log

2016-04-15T14:49:40.032 ERROR:teuthology.run_tasks:Manager failed: selinux
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 139, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/__init__.py", line 134, in __exit__
    self.teardown()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/selinux.py", line 142, in teardown
    self.get_new_denials()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/selinux.py", line 190, in get_new_denials
    denials=new_denials[remote.name])
SELinuxError: SELinux denials found on ubuntu@mira037.front.sepia.ceph.com: ['type=AVC msg=audit(1460756957.193:4372): avc:  denied  { write } for  pid=25932 comm="ms_pipe_write" laddr=172.21.5.140 lport=6789 faddr=172.21.6.116 fport=52362 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756957.951:4375): avc:  denied  { read } for  pid=25804 comm="safe_timer" name="self" dev="proc" ino=4026531841 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=lnk_file', 'type=AVC msg=audit(1460756961.212:4381): avc:  denied  { name_connect } for  pid=23955 comm="ms_pipe_write" scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756957.951:4375): avc:  denied  { search } for  pid=25804 comm="safe_timer" name="/" dev="proc" ino=1 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=dir', 'type=AVC msg=audit(1460756968.005:4383): avc:  denied  { open } for  pid=23951 comm="safe_timer" path="/var/lib/ceph/mon/ceph-mira037/store.db" dev="sda1" ino=11405255 scontext=system_u:object_r:unlabeled_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=dir', 'type=AVC msg=audit(1460756957.952:4376): avc:  denied  { read } for  pid=25804 comm="safe_timer" name="loadavg" dev="proc" ino=4026532061 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file', 'type=AVC msg=audit(1460756959.056:4377): avc:  denied  { append } for  pid=24411 comm="log" path=2F7661722F6C6F672F636570682F636570682D6D64732E6D6972613033372E6C6F67202864656C6574656429 dev="sda1" ino=9701239 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file', 'type=AVC msg=audit(1460756968.005:4383): avc:  denied  { read } for  pid=23951 comm="safe_timer" name="store.db" dev="sda1" ino=11405255 scontext=system_u:object_r:unlabeled_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=dir', 'type=AVC msg=audit(1460756957.951:4375): avc:  denied  { read } for  pid=25804 comm="safe_timer" name="status" dev="proc" ino=83579 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file', 'type=AVC msg=audit(1460756968.005:4384): avc:  denied  { getattr } for  pid=23951 comm="safe_timer" path="/var/lib/ceph/mon/ceph-mira037/store.db/000006.log" dev="sda1" ino=11405286 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file', 'type=AVC msg=audit(1460756960.348:4378): avc:  denied  { getattr } for  pid=25592 comm="osd_srv_heartbt" name="/" dev="dm-1" ino=16 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem', 'type=AVC msg=audit(1460756961.212:4382): avc:  denied  { shutdown } for  pid=23955 comm="ms_pipe_write" lport=42307 faddr=127.0.0.1 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756957.193:4371): avc:  denied  { append } for  pid=23945 comm="log" path=2F7661722F6C6F672F636570682F636570682D6D6F6E2E6D6972613033372E6C6F67202864656C6574656429 dev="sda1" ino=9701232 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file', 'type=AVC msg=audit(1460756960.348:4378): avc:  denied  { search } for  pid=25592 comm="osd_srv_heartbt" name="lib" dev="sda1" ino=9437186 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=dir', 'type=AVC msg=audit(1460756957.951:4375): avc:  denied  { search } for  pid=25804 comm="safe_timer" name="24410" dev="proc" ino=85668 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=dir', 'type=AVC msg=audit(1460756968.005:4385): avc:  denied  { getattr } for  pid=23951 comm="safe_timer" path="/var/lib/ceph/mon/ceph-mira037/store.db/LOCK" dev="sda1" ino=11405260 scontext=system_u:object_r:unlabeled_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file', 'type=AVC msg=audit(1460756957.951:4375): avc:  denied  { open } for  pid=25804 comm="safe_timer" path="/proc/24410/status" dev="proc" ino=83579 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file', 'type=AVC msg=audit(1460756961.212:4380): avc:  denied  { setopt } for  pid=23955 comm="ms_pipe_write" scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756961.212:4381): avc:  denied  { connect } for  pid=23955 comm="ms_pipe_write" scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756961.212:4379): avc:  denied  { create } for  pid=23955 comm="ms_pipe_write" scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=USER_AVC msg=audit(1460756727.594:3899): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=\'avc:  denied  { enable } for auid=1000 uid=0 gid=0 cmdline="systemctl enable ceph.target" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:init_t:s0 tclass=service  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?\'', 'type=AVC msg=audit(1460756957.193:4370): avc:  denied  { read } for  pid=25931 comm="ms_pipe_read" laddr=172.21.5.140 lport=6789 faddr=172.21.6.116 fport=52362 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=tcp_socket', 'type=AVC msg=audit(1460756957.952:4376): avc:  denied  { open } for  pid=25804 comm="safe_timer" path="/proc/loadavg" dev="proc" ino=4026532061 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=file']
2016-04-15T14:49:40.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer

Actions

Copy link

Updated by Ken Dreyer about 8 years ago

Milan, Boris, any ideas?

Actions

Copy link

Updated by Boris Ranto about 8 years ago

I suspect this will have something to do with the ~recent changes in the threading mechanism. The named threads were introduced back in December in

https://github.com/ceph/ceph/commit/4a4b447e35bbf801e7c9d5d1fadd168f394d85bd

and later updated with

https://github.com/ceph/ceph/commit/f22a09705d6b8b00ca0246f0bf9094616fc4448c

The issue seems to be that these threads do not inherit the label of the parent process. The processes should inherit the SELinux context of the parent process unless they are told otherwise. That and the fact that the threads use unlabeled_t (meaning "a generic file/exec") and not unconfined_t (meaning something like "leaving me alone, SELinux") makes it fairly weird. I'll need to dig deeper but I suspect it is related to the changes in the two referenced commits.

Actions

Copy link

Updated by Yuri Weinstein about 8 years ago

Run: http://pulpito.ceph.com/teuthology-2016-04-19_17:06:12-ceph-deploy-jewel-distro-basic-mira/
Jobs: all centos

Actions

Copy link

Updated by Boris Ranto almost 8 years ago

OK, in my testing, all of these happen when we are uninstalling the ceph packages (and SELinux policy). If the SELinux policy is uninstalled, the daemons running with that context default to unlabelled_t and this generates all of these denials (this was handled by some kernel mechanism before because we did not use named threads but the bug was still there).

The fix is to properly stop/disable the ceph daemons on uninstall. I'm already working on fix for this in wip-branto-systemd -- this packaging issue did hit us on several fronts.

Actions

Copy link