Project

General

Profile

HOWTO run integration and upgrade tests » History » Version 18

« Previous - Version 18/40 (diff) - Next » - Current version
Loïc Dachary, 05/07/2015 07:36 AM


Scheduling a suite

This requires access to a running teuthology cluster, for instance the cluster from the sepia lab.

The meaning of the teuthology-suite arguments are:

Scheduling a rados suite

NOTE: git checkout 819c63a8fa9776028311b96efe2c14d97291ff8f can be used until the code is merged. It was reverted because of a bug

In hammer rados suite generates more than 3000 jobs and it is too heavy for the purpose of validating a series of pull requests before they are merged into the stable release branch. The number of jobs can be reduced by using the --subset option. It will make sure every yaml (facet) is included in the run but will not include all the combinations. For instance:

./virtualenv/bin/teuthology-suite --subset 1/7 --priority 101 --suite rados --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports

Will reduce the number of jobs to 1/7 of the total number of jobs, as long as it includes all yaml (facets).

For a given rados suite --subset 0/7 will always create the same jobs. By running 0/7, then 1/7, etc. up to 7/7 all jobs are generated. It is a good idea to not always pick the same subset to get various combinations and increase the odds of discovering a problem.

Re-scheduling failed or dead jobs from an existing suite

  • Ask https://github.com/ceph/paddles (the server in which suite runs are stored) about the dead jobs
    run=loic-2015-03-23_01:09:31-rados-giant---basic-multi
    eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=dead | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
    

    or the fail jobs:
    run=loic-2015-03-23_01:09:31-rados-giant---basic-multi
    eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
    
  • Re-run the suite using the same command line without --filter-out and with --filter "$filter" to only schedule the jobs described in the fitler variable
    ./virtualenv/bin/teuthology-suite --filter "$filter" --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph giant
    

Killing a suite

  • if the run is scheduled but did not start yet:
    • if the suite was scheduled with --machine-type plana,burnupi,mira:
      ./virtualenv/bin/teuthology-kill -m multi -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer 
      
    • if the suite was scheduled with --machine-type vps
      ./virtualenv/bin/teuthology-kill -m vps -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer
      
  • if the run already started the -m option is not necessary
    ./virtualenv/bin/teuthology-kill -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer
    

Integration suites

Expected to be successfully run on the integration branch before asking approval to the leads (hence before asking QE to tests further)

rados
rgw
rbd
fs

QE suites

Expected to be successfully run on the release branch before passing it to the person publishing the release.

dumpling

rados
rbd
rgw
fs
ceph-deploy
upgrade/dumpling

firefly

rados
rbd
rgw
fs
krbd
kcephfs
samba
ceph-deploy
upgrade/firefly
upgrade/dumpling-firefly-x (to giant)
powercycle

giant

rados
rbd
rgw
fs
krbd
kcephfs
knfs
haddop
samba
rest
multimds
multi-version
upgrade/giant
powecycle