Feature #13031
Updated by Loïc Dachary over 8 years ago
Implement a task that builds Ceph packages and make it so they are consumed instead of the gitbuilder packages. The goal is to allow anyone with access to an OpenStack cluster to run a suite without requiring a build from the gitbuilders. In the build-packages task, tasks, the packages are built using : * "make-debs.sh":https://github.com/ceph/ceph/blob/master/make-debs.sh * make-rpm.sh And made available via an http server (package server) In both cases the output is a repository containing the packages. The build-packages tasks looks for a target with a structure identical to http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ the role *packages* and runs on it. The build-packages tasks must be run before the tasks that consume its output. This is done by modifying the interal task (see how the kernel task does it for instance): * teuthology internal tasks looks for the package-build task in the job config * if the package-build task is found, it is placed before tasks that are known to consume packages <pre> tasks: - build-packages: </pre> * when the a package-build task is run, it ** looks for found, the sha1 to be built in the *package server* ** if the package server already has the sha1, stop there ** builds the packages by locking a internal task adds an additional target of with the desired os_type, os_version and arch and running something like *packages* role <pre> mount /dev/vdb /opt ; cd /opt ; git clone + git reset sha1 roles: - [mon.0, client.0] - [packages] rm -fr /opt/Ubuntu/trusty ; sudo apt-get update ; sudo rm -fr /var/cache/apt ; sudo ln -s /opt/apt /var/cache/apt ; time\ bash -c 'sudo apt-get install -y htop ccache reprepro git ; ccache -M 5G ; export CCACHE_DIR=/opt/.ccache ; ./install-deps.sh ; NPROC=$(nproc) ./make-debs.sh /opt' </pre> ** upload * the install task looks for the *packages* role and uses the repository to it contains instead of the *package server* ** unlock the target gitbuilder * a suites/debug/build-package.yaml file is added to all ceph-qa-suite branches so that it can be append to the teuthology-suite cli and found when extracting the suite. This adds ~15 minutes to a run for a fully populated ccache on a 4 core, 16GB RAM instance. run. There are many ways to optimize this (sharing repository between jobs targeting the same branch, uploading to third party repositories, creating docker images). But these are not in the scope of this first implementation.