Project

General

Profile

Actions
Copy link

Bug #41877

closed

Tests failing with installation "Error: Package: 2:ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64 (ceph) Requires: selinux-policy-base >= 3.13.1-252.el7.1"

Added by Brad Hubbard over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
Alfredo Deza
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/nojha-2019-09-12_02:37:58-rados:singleton-master-distro-basic-smithi/4298742/

2019-09-12T03:41:49.843 INFO:teuthology.orchestra.run.smithi097.stderr:Error: Package: 2:ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64 (ceph)
2019-09-12T03:41:49.844 INFO:teuthology.orchestra.run.smithi097.stderr:           Requires: selinux-policy-base >= 3.13.1-252.el7.1

selinux-policy-base-3.13.1-252.el7.1 (metapackage) is only available in the CR channel of CentOS.

If we look at the package it seems OK.

$ rpm -qp --requires ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep selinux-policy
selinux-policy-base >= 3.13.1-229.el7_6.15

On closer inspection though we seem to be using packages from a different location than I would expect.

2019-09-12T03:41:40.059 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=81bfdf6d0f63c0f24febd6c272eaa87eb50d215b
2019-09-12T03:41:40.070 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/
2019-09-12T03:41:40.070 INFO:teuthology.task.install.rpm:Package version is 15.0.0-4863.g81bfdf6

Checking that package.

$ wget -q https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm
$ rpm -qp --requires ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep selinux-policy
selinux-policy-base >= 3.13.1-252.el7.1

Looking at the build host...

$ wget -q https://2.chacra.ceph.com/r/ceph/dmick-test-not-wp/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                              
$ rpm -qpi ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep Host                                    
Build Host  : slave-centos01                                                                              
$ rm ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                                                     
$ wget -q https://2.chacra.ceph.com/r/ceph/master/81bfdf6d0f63c0f24febd6c272eaa87eb50d215b/centos/7/flavors/default/x86_64/ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm                                         
$ rpm -qpi ceph-selinux-15.0.0-4863.g81bfdf6.el7.x86_64.rpm |grep Host                                    
Build Host  : slave-centos08

looking at the binaries from the ceph-osd packages.

default]$ eu-unstrip -n -e ./usr/bin/ceph-osd
0+0x9970af8 b87ef22a2031d0c39ea00e1facd267cbbafc2759@0x2bc ./usr/bin/ceph-osd -
dmick]$ eu-unstrip -n -e ./usr/bin/ceph-osd
0+0x9970af8 865b1a3db191e4de35f6aed1c641a1e8f71efde2@0x2bc ./usr/bin/ceph-osd -

So these are from completely different builds on different hosts. I suspect slave-centos01 has (or did have) selinux-policy-*-3.13.1-252.el7.1 installed from the CentOS CR repo at the time these packages were built.

This has affected multiple people and caused multiple teuthology runs to fail over the last few days.

Actions
Copy link
 #1

Updated by Brad Hubbard over 4 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Brad Hubbard over 4 years ago

Seems this query returns two repos with the dmick-test-not-wp repo prioritised over the default repo.

https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=81bfdf6d0f63c0f24febd6c272eaa87eb50d215b

There may be no problem with this system other than the fact one set of packages was built on a slave that had selinux-policy-*-3.13.1-252.el7.1 installed but I wonder why we would we build the same sha more than once? I also wonder why slave-centos01 is a different build environment than slave-centos08?

Actions
Copy link
 #3

Updated by Alfredo Deza over 4 years ago

  • Assignee set to Alfredo Deza

My suspicion is that some job added these repos on the fly and didn't do a proper removal. We've had some of those in the past and we've added removal steps. For example, the ganesha jobs add Ceph repos so that it can use dev versions of Ceph.

I'll make a sweep all jobs and report back if I find anything before I set out to start cleaning up affected node(s)

Actions
Copy link
 #4

Updated by Alfredo Deza over 4 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #5

Updated by Alfredo Deza over 4 years ago

  • Status changed from Fix Under Review to Resolved

The PR got merged, it will ensure that jobs that add ceph development repositories are removed after a job finishes regardless if the job passed or not.

In addition to that, I manually went over all the Jenkins slaves and removed these files so that we don't have to wait for a job that adds a shaman repo (or list file) runs in that nodes and cleans up.

Actions
Copy link
 #6

Updated by Brad Hubbard over 4 years ago

Hey Alfredo, Thanks for that. Did you check what version of the selinux-policy-* packages they were running? Adding and removing the repo is one thing but how is it handled when one of the machines adds the CR repo and upgrades to selinux-policy-*-3.13.1-252.el7.1? How do they then get downgraded to selinux-policy-*-3.13.1-229.el7_6.15 once the repo is removed? This is probably just my lack of understanding of the build system but I thought I'd ask, at least for my own illumination.

Actions
Copy link

Also available in: Atom PDF