CI

http://pulpito.ceph.com/nojha-2019-09-12_02:37:58-rados:singleton-master-distro-basic-smithi/4298742/

2019-09-12T03:41:49.843 INFO:teuthology.orchestra.run.smithi097.stderr:Error: Package: 2:ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64 (ceph)
2019-09-12T03:41:49.844 INFO:teuthology.orchestra.run.smithi097.stderr:           Requires: selinux-policy-base >= 3.13.1-252.el7.1

selinux-policy-base-3.13.1-252.el7.1 (metapackage) is only available in the CR channel of CentOS.

If we look at the package it seems OK.

$ rpm -qp --requires ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep selinux-policy
selinux-policy-base >= 3.13.1-229.el7_6.15

On closer inspection though we seem to be using packages from a different location than I would expect.

2019-09-12T03:41:40.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=81bfdf6d0f63c0f24febd6c272eaa87eb50d215b
2019-09-12T03:41:40.070 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/
2019-09-12T03:41:40.070 INFO:teuthology.task.install.rpm:Package version is 15.0.0-4863.g81bfdf6

Checking that package.

$ wget -q https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm
$ rpm -qp --requires ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep selinux-policy
selinux-policy-base >= 3.13.1-252.el7.1

Looking at the build host...

$ wget -q https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                              
$ rpm -qpi ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep Host                                    
Build Host  : slave-centos01                                                                              
$ rm ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                                                     
$ wget -q https://2.chacra.ceph.com/r/ceph/master/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                                         
$ rpm -qpi ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep Host                                    
Build Host  : slave-centos08                                                                              

looking at the binaries from the ceph-osd packages.

default]$ eu-unstrip -n -e ./usr/bin/ceph-osd
0+0x9970af8 b87ef22a2031d0c39ea00e1facd267cbbafc2759@0x2bc ./usr/bin/ceph-osd -
dmick]$ eu-unstrip -n -e ./usr/bin/ceph-osd
0+0x9970af8 865b1a3db191e4de35f6aed1c641a1e8f71efde2@0x2bc ./usr/bin/ceph-osd -

So these are from completely different builds on different hosts. I suspect slave-centos01 has (or did have) selinux-policy-*-3.13.1-252.el7.1 installed from the CentOS CR repo at the time these packages were built.

This has affected multiple people and caused multiple teuthology runs to fail over the last few days.