Ceph

Probably it just need some new rule in selinux policy. How it can be reproduced, is is part of a regular tests run?

Anyway, short info how to interpret it:

- install policycore-utils

- we can list "ceph_t" AVC rom autidt using ausearch: ausearch -m avc --raw -se ceph

- and transform it to SELinux policy rules (sometimes -R to produce macros helps but it depends):
   ausearch -m avc --raw -se ceph | audit2allow

1. echo "type=AVC msg=audit(1440160179.917:2120): avc: denied { dac_override } for pid=8523 comm="ceph-mon" capability=1 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability" |audit2allow

#============= ceph_t ==============
allow ceph_t self:capability dac_override;

(But here we should need to check why it is there, it is probably problem elsewhere - missing label or so.)

SELinux treats root as a regular user by default. This means that SELinux will deny access to a file owned by a random user with 600 file permissions to root user. The dac_override capability is used to allow root user to access these files.

An alternative solution would be to find out access to what file generates the message (this can be easily achieved if we tune up auditing rules a bit) and fix up the permissions on the file so that root user could explicitly read it.

type=SERVICE_START msg=audit(1440160179.897:2118): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="ceph-create-keys@plana63" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1440160179.898:2119): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="ceph-mon@plana63" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1440160179.917:2120): avc:  denied  { dac_override } for  pid=8523 comm="ceph-mon" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability
type=SYSCALL msg=audit(1440160179.917:2120): arch=c000003e syscall=2 success=yes exit=3 a0=7fc661101c98 a1=441 a2=1a4 a3=ffffffff items=0 ppid=1 pid=8523 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-mon" exe="/usr/bin/ceph-mon" subj=system_u:system_r:ceph_t:s0 key=(null)
type=SERVICE_STOP msg=audit(1440160179.928:2121): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=' comm="ceph-mon@plana63" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

hmm. misc.log has

2015-08-21T08:29:39.919450-04:00 plana63 ceph-mon: 2015-08-21 08:29:39.918218 7fc65dccf880 -1 did not load config file, using default settings.
2015-08-21T08:29:39.919985-04:00 plana63 ceph-mon: monitor data directory at '/var/lib/ceph/mon/ceph-plana63' does not exist: have you run 'mkfs'?
2015-08-21T08:29:39.929505-04:00 plana63 systemd: ceph-mon@plana63.service: main process exited, code=exited, status=1/FAILURE
2015-08-21T08:29:39.930078-04:00 plana63 systemd: Unit ceph-mon@plana63.service entered failed state.

but at that point hte package was only just getting installed... wth. any idea what would be trying to start the service at this point?

...
2015-08-21T08:29:10.538505-04:00 plana63 yum[8478]: Installed: 1:libcephfs1-devel-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:10.897954-04:00 plana63 yum[8478]: Installed: 1:libradosstriper1-devel-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:11.978002-04:00 plana63 yum[8478]: Installed: 1:librbd1-devel-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:17.365482-04:00 plana63 yum[8478]: Installed: 1:ceph-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:17.719963-04:00 plana63 systemd: Stopping ceph target allowing to start/stop all ceph*@.service instances at once.
2015-08-21T08:29:17.720336-04:00 plana63 systemd: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once.
2015-08-21T08:29:22.516075-04:00 plana63 sshd[8502]: Connection closed by 10.214.140.231 [preauth]
2015-08-21T08:29:39.830461-04:00 plana63 dbus-daemon: dbus[648]: avc:  received policyload notice (seqno=3)
2015-08-21T08:29:39.831259-04:00 plana63 dbus[648]: avc:  received policyload notice (seqno=3)
2015-08-21T08:29:39.832439-04:00 plana63 dbus-daemon: dbus[648]: [system] Reloaded configuration
2015-08-21T08:29:39.832834-04:00 plana63 dbus[648]: [system] Reloaded configuration
2015-08-21T08:29:39.898248-04:00 plana63 systemd: Starting Ceph cluster key creator task...
2015-08-21T08:29:39.899719-04:00 plana63 systemd: Started Ceph cluster key creator task.
2015-08-21T08:29:39.900274-04:00 plana63 systemd: Starting Ceph cluster monitor daemon...
2015-08-21T08:29:39.900748-04:00 plana63 systemd: Started Ceph cluster monitor daemon.
2015-08-21T08:29:39.901173-04:00 plana63 systemd: Starting ceph target allowing to start/stop all ceph*@.service instances at once.
2015-08-21T08:29:39.901681-04:00 plana63 systemd: Reached target ceph target allowing to start/stop all ceph*@.service instances at once.
2015-08-21T08:29:39.907387-04:00 plana63 yum[8478]: Installed: 1:ceph-selinux-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:39.919450-04:00 plana63 ceph-mon: 2015-08-21 08:29:39.918218 7fc65dccf880 -1 did not load config file, using default settings.
2015-08-21T08:29:39.919985-04:00 plana63 ceph-mon: monitor data directory at '/var/lib/ceph/mon/ceph-plana63' does not exist: have you run 'mkfs'?
2015-08-21T08:29:39.929505-04:00 plana63 systemd: ceph-mon@plana63.service: main process exited, code=exited, status=1/FAILURE
2015-08-21T08:29:39.930078-04:00 plana63 systemd: Unit ceph-mon@plana63.service entered failed state.
2015-08-21T08:29:40.077370-04:00 plana63 ceph-create-keys: admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
2015-08-21T08:29:40.084631-04:00 plana63 ceph-create-keys: INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
2015-08-21T08:29:40.699362-04:00 plana63 yum[8478]: Installed: 1:libcephfs_jni1-9.0.2-1536.g10f6b64.el7.x86_64
2015-08-21T08:29:41.030152-04:00 plana63 yum[8478]: Installed: 1:libcephfs_jni1-devel-9.0.2-1536.g10f6b64.el7.x86_64
...

oh, it's in ceph.spec:

%post selinux
/sbin/service ceph stop >/dev/null 2>&1
semodule -n -i %{_datadir}/selinux/packages/ceph.pp
if /usr/sbin/selinuxenabled ; then
    /usr/sbin/load_policy
    %relabel_files
fi
/sbin/service ceph start >/dev/null 2>&1
exit 0

%postun selinux
if [ $1 -eq 0 ]; then
    /sbin/service ceph stop >/dev/null 2>&1
    semodule -n -r ceph
    if /usr/sbin/selinuxenabled ; then
       /usr/sbin/load_policy
       %relabel_files
    fi;
    /sbin/service ceph start >/dev/null 2>&1
fi;
exit 0

as a rule we never start or stop services during package install or upgrade. can change this so any changes take effect on the next restart?

We need to stop/start the service in order to get its SELinux context fixed when installing the policy.

btw: There were some improvements to this that already got merged upstream. It would be nice to start the service only if it was running before, though. I'll send out a PR to improve that.

The PR for the safer process restart procedure:

https://github.com/ceph/ceph/pull/5674