Feature #13031
Updated by Loïc Dachary over 8 years ago
* teuthology https://github.com/ceph/teuthology/pull/623
* ceph-qa-suite https://github.com/ceph/ceph-qa-suite/pull/565
Implement a task that builds Ceph packages and make it so they are consumed instead of the gitbuilder packages. The goal is to allow anyone with access to an OpenStack cluster to run a suite without requiring a build from the gitbuilders.
In the build-packages task, the packages are built using :
* "make-debs.sh":https://github.com/ceph/ceph/blob/master/make-debs.sh
* make-rpm.sh
And made available via an http server (package server) with a structure identical to http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/
The build-packages tasks must be run before the tasks that consume its output. This is done by modifying the interal task (see how the kernel task does it for instance):
* teuthology internal tasks looks for the package-build task in the job config
* if the package-build task is found, it is placed before tasks that are known to consume packages
<pre>
tasks:
- build-packages:
</pre>
* when the package-build task is run, it
** looks for the sha1 to be built in the *package server*
** if the package server already has the sha1, stop there
** builds the packages by locking a target of the desired os_type, os_version and arch and running something like
<pre>
mount /dev/vdb /opt ; cd /opt ; git clone + git reset sha1
rm -fr /opt/Ubuntu/trusty ; sudo apt-get update ; sudo rm -fr /var/cache/apt ; sudo ln -s /opt/apt /var/cache/apt ; time\
bash -c 'sudo apt-get install -y htop ccache reprepro git ; ccache -M 5G ; export CCACHE_DIR=/opt/.ccache ; ./install-deps.sh ; NPROC=$(nproc) ./make-debs.sh /opt'
</pre>
** upload the repository to the *package server*
** unlock the target
* a suites/debug/build-package.yaml file is added to all ceph-qa-suite branches so that it can be append to the teuthology-suite cli and found when extracting the suite.
This adds ~15 minutes to a run for a fully populated ccache on a 4 core, 16GB RAM instance.
There are many ways to optimize this (sharing repository between jobs targeting the same branch, uploading to third party repositories, creating docker images). But these are not in the scope of this first implementation.