Ceph

SELinuxError: SELinux denials found on ubuntu@vpm130.front.sepia.ceph.com: ['type=AVC msg=audit(1454631049.211:348): avc:  denied  { ioctl } for  pid=7958 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/kern.log" dev="vda1" ino=184549511 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454631040.829:307): avc:  denied  { search } for  pid=7958 comm=72733A6D61696E20513A526567 name="cephtest" dev="vda1" ino=159383848 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir', 'type=AVC msg=audit(1454633044.929:3772): avc:  denied  { read } for  pid=21665 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=86917 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file', 'type=AVC msg=audit(1454632968.211:3627): avc:  denied  { read } for  pid=19972 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=86917 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file', 'type=AVC msg=audit(1454631040.829:306): avc:  denied  { search } for  pid=7958 comm=72733A6D61696E20513A526567 name="cephtest" dev="vda1" ino=159383848 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir', 'type=AVC msg=audit(1454633046.524:3780): avc:  denied  { read } for  pid=21833 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=86917 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file', 'type=AVC msg=audit(1454631040.854:312): avc:  denied  { ioctl } for  pid=7958 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/misc.log" dev="vda1" ino=184549512 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454631730.990:2133): avc:  denied  { ioctl } for  pid=12525 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/kern.log" dev="vda1" ino=184549511 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454631040.854:311): avc:  denied  { open } for  pid=7958 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/misc.log" dev="vda1" ino=184549512 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454633048.348:3782): avc:  denied  { getattr } for  pid=22015 comm="ceph-osd" path="/dev/sr0" dev="devtmpfs" ino=8749 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:removable_device_t:s0 tclass=blk_file', 'type=AVC msg=audit(1454632967.188:3624): avc:  denied  { dac_override } for  pid=19864 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454631040.485:293): avc:  denied  { search } for  pid=7958 comm=72733A6D61696E20513A526567 name="cephtest" dev="vda1" ino=159383848 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir', 'type=AVC msg=audit(1454631657.447:2125): avc:  denied  { ioctl } for  pid=12525 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/misc.log" dev="vda1" ino=184549512 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454631040.485:294): avc:  denied  { search } for  pid=7958 comm=72733A6D61696E20513A526567 name="cephtest" dev="vda1" ino=159383848 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir', 'type=AVC msg=audit(1454631049.211:347): avc:  denied  { open } for  pid=7958 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/kern.log" dev="vda1" ino=184549511 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454632971.258:3634): avc:  denied  { getattr } for  pid=20305 comm="ceph-osd" path="/dev/sr0" dev="devtmpfs" ino=8749 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:removable_device_t:s0 tclass=blk_file', 'type=AVC msg=audit(1454632969.542:3632): avc:  denied  { read } for  pid=20090 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=86917 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file', 'type=AVC msg=audit(1454632971.385:3635): avc:  denied  { dac_override } for  pid=20353 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454631040.829:305): avc:  denied  { search } for  pid=7958 comm=72733A6D61696E20513A526567 name="cephtest" dev="vda1" ino=159383848 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir', 'type=AVC msg=audit(1454631730.990:2132): avc:  denied  { open } for  pid=12525 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/kern.log" dev="vda1" ino=184549511 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454633045.627:3777): avc:  denied  { dac_override } for  pid=21709 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454632981.199:3759): avc:  denied  { dac_override } for  pid=20811 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext
=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454632980.825:3756): avc:  denied  { dac_override } for  pid=20786 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454633048.389:3783): avc:  denied  { dac_override } for  pid=22052 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability', 'type=AVC msg=audit(1454631657.447:2124): avc:  denied  { open } for  pid=12525 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/misc.log" dev="vda1" ino=184549512 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file', 'type=AVC msg=audit(1454632903.971:3609): avc:  denied  { dac_override } for  pid=19076 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability']

http://qa-proxy.ceph.com/teuthology/vasu-2016-02-04_16:06:51-selinux-jewel---basic-vps/7190/teuthology.log

hopefully this format is better, some tag above is messing up the format,

copied from one of the audit logs

[vakulkar@vakulkar ~]$ grep 'ceph-osd\|ceph-mon' audit.log
type=SERVICE_START msg=audit(1454632668.606:3935): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ceph-mon@vpm193 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1454632750.313:4003): avc:  denied  { dac_override } for  pid=22477 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability
type=SYSCALL msg=audit(1454632750.313:4003): arch=c000003e syscall=2 success=yes exit=3 a0=7f5da5c18b18 a1=441 a2=1a4 a3=0 items=0 ppid=22462 pid=22477 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632815.209:4028): avc:  denied  { read } for  pid=22848 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=95548 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file
type=SYSCALL msg=audit(1454632815.209:4028): arch=c000003e syscall=59 success=yes exit=0 a0=9cb9a0 a1=9b5590 a2=7ffeccb6ca70 a3=7ffeccb6a740 items=0 ppid=22757 pid=22848 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632815.990:4031): avc:  denied  { dac_override } for  pid=22899 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability
type=SYSCALL msg=audit(1454632815.990:4031): arch=c000003e syscall=2 success=yes exit=3 a0=7f71fd644b18 a1=441 a2=1a4 a3=0 items=0 ppid=22887 pid=22899 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=SERVICE_START msg=audit(1454632818.319:4036): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ceph-osd@0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1454632818.748:4037): avc:  denied  { getattr } for  pid=23161 comm="ceph-osd" path="/dev/sr0" dev="devtmpfs" ino=8707 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:removable_device_t:s0 tclass=blk_file
type=SYSCALL msg=audit(1454632818.748:4037): arch=c000003e syscall=4 success=yes exit=0 a0=7f945b50c480 a1=7f94356a8870 a2=7f94356a8870 a3=b00 items=0 ppid=1 pid=23161 auid=4294967295 uid=167 gid=167 euid=167 suid=167 fsuid=167 egid=167 sgid=167 fsgid=167 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632826.284:4060): avc:  denied  { dac_override } for  pid=23462 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability
type=SYSCALL msg=audit(1454632826.284:4060): arch=c000003e syscall=2 success=yes exit=3 a0=7f3457c0eb18 a1=441 a2=1a4 a3=0 items=0 ppid=23447 pid=23462 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632891.385:4079): avc:  denied  { read } for  pid=24311 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=95548 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file
type=SYSCALL msg=audit(1454632891.385:4079): arch=c000003e syscall=59 success=yes exit=0 a0=27ab930 a1=278d0c0 a2=7ffc15fcf3e0 a3=7ffc15fcd0b0 items=0 ppid=24220 pid=24311 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632892.683:4086): avc:  denied  { read } for  pid=24425 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=95548 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file
type=SYSCALL msg=audit(1454632892.683:4086): arch=c000003e syscall=59 success=yes exit=0 a0=2545bf0 a1=25418d0 a2=7ffccdda65e0 a3=7ffccdda42b0 items=0 ppid=24333 pid=24425 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=SERVICE_START msg=audit(1454632894.356:4087): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ceph-osd@1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1454632894.673:4088): avc:  denied  { dac_override } for  pid=24668 comm="ceph-osd" capability=1  scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability
type=SYSCALL msg=audit(1454632894.673:4088): arch=c000003e syscall=2 success=yes exit=3 a0=7fdd9951eb18 a1=441 a2=1a4 a3=0 items=0 ppid=24662 pid=24668 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1454632894.680:4089): avc:  denied  { getattr } for  pid=24636 comm="ceph-osd" path="/dev/sr0" dev="devtmpfs" ino=8707 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:removable_device_t:s0 tclass=blk_file
type=SYSCALL msg=audit(1454632894.680:4089): arch=c000003e syscall=4 success=yes exit=0 a0=7f2176102420 a1=7f214eda5870 a2=7f214eda5870 a3=b00 items=0 ppid=1 pid=24636 auid=4294967295 uid=167 gid=167 euid=167 suid=167 fsuid=167 egid=167 sgid=167 fsgid=167 tty=(none) ses=4294967295 comm="ceph-osd" exe="/usr/bin/ceph-osd" subj=system_u:system_r:ceph_t:s0 key=(null)

It looks like there is a lot of stuff happening, here:

Some of the processes are mislabelled. This suggests that either the policy is not installed or (more likely) the processes were not restarted after the policy was activated -- this can happen if the processes were started outside the systemd environment which, afaik, teuthology does.

Then, you are probably hitting this:

http://tracker.ceph.com/issues/12755

I believe we hit this when we are manually restarting the services.

Also, you are hitting

type=AVC msg=audit(1454638638.323:4189): avc: denied { read } for pid=24231 comm="ceph-osd" path="/run/lock/ceph-disk" dev="tmpfs" ino=95162 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:var_lock_t:s0 tclass=file

which looks like something that we might want to add to the policy and

type=AVC msg=audit(1454638640.317:4191): avc: denied { getattr } for pid=24419 comm="ceph-osd" path="/dev/sr0" dev="devtmpfs" ino=8683 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:removable_device_t:s0 tclass=blk_file

which looks pretty strange to me as it suggests that ceph-osd is trying to access /dev/sr0. I fail to see valid use case when ceph-osd would need to access /dev/sr0. Maybe, it is part of one of the tests to do that -- i.e. tell ceph-osd that /dev/sr0 is its regular storage to see how it can handle it?

I can also see few denials like

type=AVC msg=audit(1454636765.427:371): avc: denied { open } for pid=7987 comm=72733A6D61696E20513A526567 path="/home/ubuntu/cephtest/archive/syslog/misc.log" dev="vda1" ino=184549512 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

which suggest a test suite problem -- you are storing your logs in a directory that was not designed for that so syslog can't access them.

TLDR:
Essentially, every denial that writes something about cephtest or have a dir/file in /home/ubuntu is not related to the ceph policy. A denial that shows one of ceph-* daemons with scontext unlabeled_t says that the policy is not loaded or the daemon was not restarted after the policy was loaded. The dac_override denials suggest that we are running under root for too often. The '/dev/sr0' denials probably mean a misconfiguration (albeit it might be intentional misconfiguration in this case). The denials that deny read on /run/lock/ceph-disk could be something we need to add to the SELinux policy.

Policy is installed as i see in the logs

2016-02-04T18:10:37.548 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Running transaction check
2016-02-04T18:10:37.548 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] ---> Package ceph.x86_64 1:10.0.2-1002.gd3925a5.el7 will be installed
2016-02-04T18:10:37.549 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: python-rbd = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.647 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: python-rados = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.647 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: python-cephfs = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.659 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: librbd1 = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.659 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: librados2 = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.660 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: libcephfs1 = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64
2016-02-04T18:10:37.660 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm176][DEBUG ] --> Processing Dependency: ceph-selinux = 1:10.0.2-1002.gd3925a5.el7 for package: 1:ceph-10.0.2-1002.gd3925a5.el7.x86_64

Also the process is not started by teuthology it uses the init as invoked by ceph-deploy, you can check the logs as pasted in comment 3, also pasting few lines below

2016-02-04T18:15:44.679 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm097][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.vpm097 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-vpm097/keyring
2016-02-04T18:15:45.015 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm097][INFO  ] Running command: sudo systemctl enable ceph-mds@vpm097
2016-02-04T18:15:45.051 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm097][WARNING] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@vpm097.service to /usr/lib/systemd/system/ceph-mds@.service.
2016-02-04T18:15:45.170 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm097][INFO  ] Running command: sudo systemctl start ceph-mds@vpm097
2016-02-04T18:15:45.259 INFO:teuthology.orchestra.run.vpm176.stderr:[vpm097][INFO  ] Running command: sudo systemctl enable ceph.target

I agree the /home/ubuntu/cephtest denails is not an issue, I think its something teuthology should take care of or just filter out the denails, There is a tracker for that already.

Yes, the policy is installed in both cases. However, there is an issue with the way daemons are being started/stopped/restarted. See the logs

2016-02-04T18:21:04.389 INFO:tasks.ceph_deploy:Stopping ceph...
2016-02-04T18:21:04.390 INFO:teuthology.orchestra.run.vpm097:Running: 'sudo stop ceph-all || sudo service ceph stop || sudo systemctl stop ceph.target'
2016-02-04T18:21:04.444 INFO:teuthology.orchestra.run.vpm097.stderr:sudo: stop: command not found
2016-02-04T18:21:04.467 INFO:teuthology.orchestra.run.vpm097.stderr:Redirecting to /bin/systemctl stop  ceph.service
2016-02-04T18:21:04.471 INFO:teuthology.orchestra.run.vpm097.stderr:Failed to stop ceph.service: Unit ceph.service not loaded.
2016-02-04T18:21:04.484 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo stop ceph-all || sudo service ceph stop || sudo systemctl stop ceph.target'
2016-02-04T18:21:04.546 INFO:teuthology.orchestra.run.vpm176.stderr:sudo: stop: command not found
2016-02-04T18:21:04.575 INFO:teuthology.orchestra.run.vpm176.stderr:Redirecting to /bin/systemctl stop  ceph.service
2016-02-04T18:21:04.579 INFO:teuthology.orchestra.run.vpm176.stderr:Failed to stop ceph.service: Unit ceph.service not loaded.
2016-02-04T18:21:04.594 INFO:teuthology.orchestra.run.vpm199:Running: 'sudo stop ceph-all || sudo service ceph stop || sudo systemctl stop ceph.target'
2016-02-04T18:21:04.645 INFO:teuthology.orchestra.run.vpm199.stderr:sudo: stop: command not found
2016-02-04T18:21:04.670 INFO:teuthology.orchestra.run.vpm199.stderr:Redirecting to /bin/systemctl stop  ceph.service
2016-02-04T18:21:04.674 INFO:teuthology.orchestra.run.vpm199.stderr:Failed to stop ceph.service: Unit ceph.service not loaded.
2016-02-04T18:21:04.689 INFO:teuthology.orchestra.run.vpm097:Running: 'sudo status ceph-all || sudo service ceph status || sudo systemctl status ceph.target'
2016-02-04T18:21:04.741 INFO:teuthology.orchestra.run.vpm097.stderr:sudo: status: command not found
2016-02-04T18:21:04.762 INFO:teuthology.orchestra.run.vpm097.stderr:Redirecting to /bin/systemctl status  ceph.service
2016-02-04T18:21:04.765 INFO:teuthology.orchestra.run.vpm097.stdout:â— ceph.service
2016-02-04T18:21:04.765 INFO:teuthology.orchestra.run.vpm097.stdout:   Loaded: not-found (Reason: No such file or directory)
2016-02-04T18:21:04.765 INFO:teuthology.orchestra.run.vpm097.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.777 INFO:teuthology.orchestra.run.vpm097.stdout:â— ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
2016-02-04T18:21:04.778 INFO:teuthology.orchestra.run.vpm097.stdout:   Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: disabled)
2016-02-04T18:21:04.778 INFO:teuthology.orchestra.run.vpm097.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.778 INFO:teuthology.orchestra.run.vpm097.stdout:
2016-02-04T18:21:04.778 INFO:teuthology.orchestra.run.vpm097.stdout:Feb 05 02:21:04 vpm097 systemd[1]: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once.
2016-02-04T18:21:04.779 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo status ceph-all || sudo service ceph status || sudo systemctl status ceph.target'
2016-02-04T18:21:04.832 INFO:teuthology.orchestra.run.vpm176.stderr:sudo: status: command not found
2016-02-04T18:21:04.851 INFO:teuthology.orchestra.run.vpm176.stderr:Redirecting to /bin/systemctl status  ceph.service
2016-02-04T18:21:04.857 INFO:teuthology.orchestra.run.vpm176.stdout:â— ceph.service
2016-02-04T18:21:04.857 INFO:teuthology.orchestra.run.vpm176.stdout:   Loaded: not-found (Reason: No such file or directory)
2016-02-04T18:21:04.857 INFO:teuthology.orchestra.run.vpm176.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.869 INFO:teuthology.orchestra.run.vpm176.stdout:â— ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
2016-02-04T18:21:04.869 INFO:teuthology.orchestra.run.vpm176.stdout:   Loaded: loaded (/usr/lib/systemd/system/ceph.target; enabled; vendor preset: disabled)
2016-02-04T18:21:04.869 INFO:teuthology.orchestra.run.vpm176.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.869 INFO:teuthology.orchestra.run.vpm176.stdout:
2016-02-04T18:21:04.869 INFO:teuthology.orchestra.run.vpm176.stdout:Feb 05 02:21:04 vpm176 systemd[1]: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once.
2016-02-04T18:21:04.870 INFO:teuthology.orchestra.run.vpm199:Running: 'sudo status ceph-all || sudo service ceph status || sudo systemctl status ceph.target'
2016-02-04T18:21:04.922 INFO:teuthology.orchestra.run.vpm199.stderr:sudo: status: command not found
2016-02-04T18:21:04.942 INFO:teuthology.orchestra.run.vpm199.stderr:Redirecting to /bin/systemctl status  ceph.service
2016-02-04T18:21:04.946 INFO:teuthology.orchestra.run.vpm199.stdout:â— ceph.service
2016-02-04T18:21:04.946 INFO:teuthology.orchestra.run.vpm199.stdout:   Loaded: not-found (Reason: No such file or directory)
2016-02-04T18:21:04.946 INFO:teuthology.orchestra.run.vpm199.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.960 INFO:teuthology.orchestra.run.vpm199.stdout:â— ceph.target - ceph target allowing to start/stop all ceph*@.service instances at once
2016-02-04T18:21:04.960 INFO:teuthology.orchestra.run.vpm199.stdout:   Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor preset: disabled)
2016-02-04T18:21:04.960 INFO:teuthology.orchestra.run.vpm199.stdout:   Active: inactive (dead)
2016-02-04T18:21:04.960 INFO:teuthology.orchestra.run.vpm199.stdout:
2016-02-04T18:21:04.960 INFO:teuthology.orchestra.run.vpm199.stdout:Feb 05 02:21:04 vpm199 systemd[1]: Stopped target ceph target allowing to start/stop all ceph*@.service instances at once.
2016-02-04T18:21:04.961 INFO:teuthology.orchestra.run.vpm097:Running: 'sudo ps aux | grep -v grep | grep ceph'
2016-02-04T18:21:05.018 INFO:teuthology.orchestra.run.vpm097.stdout:ceph     27119  0.3  1.4 350432 26536 ?        Ssl  02:15   0:01 /usr/bin/ceph-mon -f --cluster ceph --id vpm097 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.019 INFO:teuthology.orchestra.run.vpm097.stdout:ceph     27680  0.0  0.7 324636 12460 ?        Ssl  02:15   0:00 /usr/bin/ceph-mds -f --cluster ceph --id vpm097 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.019 INFO:teuthology.orchestra.run.vpm097.stdout:ceph     29823  1.5  2.0 856588 37112 ?        Ssl  02:18   0:02 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.019 INFO:teuthology.orchestra.run.vpm097.stdout:ceph     31672  3.2  2.8 856336 50004 ?        Ssl  02:19   0:02 /usr/bin/ceph-osd -f --cluster ceph --id 3 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.019 INFO:teuthology.orchestra.run.vpm176:Running: 'sudo ps aux | grep -v grep | grep ceph'
2016-02-04T18:21:05.085 INFO:teuthology.orchestra.run.vpm176.stdout:ceph     21497  0.2  1.2 336048 23104 ?        Ssl  02:15   0:00 /usr/bin/ceph-mon -f --cluster ceph --id vpm176 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.085 INFO:teuthology.orchestra.run.vpm176.stdout:ceph     21985  0.0  0.5 355392 10660 ?        Ssl  02:15   0:00 /usr/bin/ceph-mds -f --cluster ceph --id vpm176 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.086 INFO:teuthology.orchestra.run.vpm176.stdout:ceph     22914  0.8  2.0 855048 36236 ?        Ssl  02:16   0:02 /usr/bin/ceph-osd -f --cluster ceph --id 0 --setuser ceph --setgroup ceph
2016-02-04T18:21:05.086 INFO:teuthology.orchestra.run.vpm176.stdout:ceph     24359  1.4  2.1 858636 38668 ?        Ssl  02:17   0:03 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

This suggests that teuthology failed to stop the ceph daemons so it must have failed to restart them when ceph-selinux was being installed so we end up running the daemons with unconfined_t instead of ceph_t -> all the unconfined_t denials. Hence, the issue is that systemctl cannot stop/start the daemons. This usually happens when you don't manage the daemons via systemd but manually - it is not the only reason why this can happen but none of those should have anything to do with Ceph SELinux policy.

Boris, you are looking at the end of the logs, I think during the test end it does try to stop those daemons as you have seen below, I can explain how it works if you can ping me on irc.
There could be an issue with how ceph-deploy is starting it which we can go through logs or live system.

Boris,

you can ignore the unlabeled in the above logs for now, I will run this on non vm' system to check what path refers in logs for vda1 device

   [''type=AVC msg=audit(1455256515.133:4565): avc:  denied  { append } for  pid=30994
    comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.508:4725): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256514.860:4355): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.801:4989): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.495:4586): avc:  denied  { shutdown
    } for  pid=32710 comm="ceph-osd" laddr=172.21.2.87 lport=60417 faddr=172.21.2.64
    fport=6803 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=tcp_socket'', ''type=AVC msg=audit(1455256508.430:3994): avc:  denied  {
    open } for  pid=27037 comm="ceph-mon" path="/var/lib/ceph/mon/ceph-vpm087/store.db" 
    dev="vda1" ino=268506178 scontext=system_u:object_r:unlabeled_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0
    tclass=dir'', ''type=AVC msg=audit(1455256515.799:4908): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.133:4534): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.798:4881): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256514.859:4331): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256514.860:4350): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256514.859:4330): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
    tclass=file'', ''type=AVC msg=audit(1455256515.131:4478): avc:  denied  { append
    } for  pid=30994 comm="ceph-osd" path=2F7661722F6C6F672F636570682F636570682D6F73642E312E6C6F67202864656C6574656429
    dev="vda1" ino=285306848 scontext=system_u:object_r:unlabeled_t:s0 tcontext=system_u:object_r:unlabeled_t:s0

Boris,

Other than the dac_overide and /run/lock/ceph-disk, this is one additional thing that should be fixed, let me know if you are fixing those

SELinux denials found on ubuntu@mira121.front.sepia.ceph.com: ['type=USER_AVC msg=audit(1456246476.722:3031): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=\'avc: denied { enable } for auid=1000 uid=0 gid=0 cmdline="systemctl enable ceph.target" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:init_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?\''] 

Vasu,

yep, I plan to fix these (at least the /run/lock and dac_override issue). I do have some clues as to have to proceed but I did not have all that much time to fix them, yet.

As for the latest issue that you mentioned, a brief look suggests that the systemctl or /usr/lib/systemd/systemd binary is mislabelled. That system should hit that whenever it tries to enable a system service/target, not just ceph.target. Was the system restarted (it is the pid 1 that is mislabelled) and fully relabelled after SELinux was made active?

That sounds interesting, The system is not restarted nor relabelled, other than whatever ceph-deploy does internally to bring up the cluster. I can ignore this in our tests if its not a ceph issue but just wanted to bring it up and probably raise a bz for systemd folks since it seems like an issue with enable.

Here are the 7 tests that reported this same issue.
http://pulpito.ceph.com/vasu-2016-02-22_21:11:55-selinux-jewel---basic-multi/

Vase, these should hopefully be fixed in latest master, can you re-run the tests?

Boris, Thanks will schedule the tests and update here.

I think this issue is fixed.

I am only seeing one below related to ceph.target and I think i will just ignore that since you told before its a systemd issue, there have been some other issues in this run http://pulpito.ceph.com/vasu-2016-03-15_15:34:41-selinux-master---basic-mira/
but I dont think anything is related to ceph.

SELinux denials found on ubuntu@mira121.front.sepia.ceph.com: ['type=USER_AVC msg=audit(1458092966.151:3237): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=\'avc: denied { enable } for auid=1000 uid=0 gid=0 cmdline="systemctl enable ceph.target" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=system_u:system_r:init_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?\''] 

I still wonder about the systemctl denial. I am not currently aware of any way that could be caused by our SELinux policy but just to be sure: Can you send mr the output of

• ps -AZ | egrep 'systemd$'

on any of the machines that is hitting the systemctl denial?

Boris, missed your update on this ticket, I will get that info, I have also raised a bz for systemd selinux policy
https://bugzilla.redhat.com/show_bug.cgi?id=1319871