Project

General

Profile

HOWTO run integration and upgrade tests » History » Version 15

Loïc Dachary, 05/04/2015 03:22 PM
don't use --owner

1 3 Loïc Dachary
h3. Scheduling a suite
2 3 Loïc Dachary
3 3 Loïc Dachary
This requires access to a running teuthology cluster, for instance the cluster from the sepia lab.
4 3 Loïc Dachary
5 3 Loïc Dachary
* git clone https://github.com/ceph/teuthology/
6 3 Loïc Dachary
* cd teuthology
7 3 Loïc Dachary
* ./bootstrap
8 8 Loïc Dachary
* Test the suite with *--dry-run*, i.e. something like:
9 3 Loïc Dachary
<pre>
10 15 Loïc Dachary
./virtualenv/bin/teuthology-suite --dry-run --filter-out btrfs,ext4 --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph giant
11 1 Loïc Dachary
</pre>
12 6 Loïc Dachary
* Review the jobs to be scheduled and if it matches what is expected, run the same command without the *--dry-run*
13 9 Loïc Dachary
* Assuming the suite was run on the sepia lab, it will immediately show up at http://pulpito.ceph.com/ (for instance http://pulpito.ceph.com/loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer---basic-multi/). Note the *loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer---basic-multi* part of the path: it matches the run name displayed by teuthology-suite
14 3 Loïc Dachary
15 6 Loïc Dachary
The meaning of the teuthology-suite arguments are:
16 4 Loïc Dachary
17 4 Loïc Dachary
* *--suite*  a reference to https://github.com/ceph/ceph-qa-suite/tree/master/suites. For instance --suite rados means run all jobs at https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados
18 4 Loïc Dachary
* *--suite-branch* a reference to the ceph-qa-suite branch to use. For instance --suite rados --suite-branch giant means run all jobs at https://github.com/ceph/ceph-qa-suite/tree/giant/suites/rados instead of https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados
19 4 Loïc Dachary
* *--priority* is the the priority of the job in the queue (lower numbers are higher priority). By default it is 1000 and if nothing is urgent it should be used. If in the process of debugging a single job that fails, the priority 101 can be used so that it is scheduled faster.
20 4 Loïc Dachary
* *--machine-type* is the kind of machine that will be provisionned to run the tests. Use *plana,burnupi,mira* for bare metal or *vps* for virtual machines.
21 4 Loïc Dachary
* *--ceph* is the branch of the Ceph repository to use and defaults to master. For instance --ceph giant means use https://github.com/ceph/ceph/tree/giant/ instead of https://github.com/ceph/ceph/tree/master/
22 4 Loïc Dachary
* *--email* when the run is complete an email will be sent to this address with a summary of the results.
23 4 Loïc Dachary
24 14 Loïc Dachary
h3. Scheduling a suite without error injection
25 13 Loïc Dachary
26 13 Loïc Dachary
In hammer "rados suite":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados generates more than a thousand jobs and it is too heavy for the purpose of validating a series of pull requests before they are merged into the stable release branch. The number of jobs can be reduced by:
27 13 Loïc Dachary
28 13 Loïc Dachary
* not test btrfs and ext4 so only xfs remains ( --filter-out btrfs,ext4 )
29 13 Loïc Dachary
* removing the thrash and msg-failures yaml files and only keep the workload that verifies a run produces the expected result (see for instance "this branch for the hammer rados suite":https://github.com/dachary/ceph-qa-suite/tree/wip-rados-no-thrash-hammer)
30 13 Loïc Dachary
31 1 Loïc Dachary
For instance, the following
32 13 Loïc Dachary
<pre>
33 15 Loïc Dachary
./virtualenv/bin/teuthology-suite --dry-run --priority 1000 --suite rados --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports
34 13 Loïc Dachary
</pre>
35 1 Loïc Dachary
generated 3224 jobs but the following
36 13 Loïc Dachary
<pre>
37 13 Loïc Dachary
git clone -b wip-rados-no-thrash-hammer https://github.com/dachary/ceph-qa-suite.git /tmp/ceph-qa-suite
38 15 Loïc Dachary
./virtualenv/bin/teuthology-suite --filter-out btrfs,ext4 --priority 1000 --suite rados --suite-dir /tmp/ceph-qa-suite --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports
39 13 Loïc Dachary
</pre>
40 13 Loïc Dachary
generated 122 jobs.
41 13 Loïc Dachary
42 3 Loïc Dachary
h3. Re-scheduling failed or dead jobs from an existing suite
43 3 Loïc Dachary
44 3 Loïc Dachary
* Ask https://github.com/ceph/paddles (the server in which suite runs are stored) about the *dead* jobs
45 3 Loïc Dachary
<pre>
46 3 Loïc Dachary
run=loic-2015-03-23_01:09:31-rados-giant---basic-multi
47 3 Loïc Dachary
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=dead | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
48 3 Loïc Dachary
</pre>
49 5 Loïc Dachary
or the *fail* jobs:
50 3 Loïc Dachary
<pre>
51 3 Loïc Dachary
run=loic-2015-03-23_01:09:31-rados-giant---basic-multi
52 1 Loïc Dachary
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
53 3 Loïc Dachary
</pre>
54 3 Loïc Dachary
* Re-run the suite using the same command line without *--filter-out* and with *--filter "$filter"* to only schedule the jobs described in the *fitler* variable
55 3 Loïc Dachary
<pre>
56 15 Loïc Dachary
./virtualenv/bin/teuthology-suite --filter "$filter" --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph giant
57 3 Loïc Dachary
</pre>
58 3 Loïc Dachary
59 7 Loïc Dachary
h3. Killing a suite
60 7 Loïc Dachary
61 7 Loïc Dachary
* if the run is scheduled but did not start yet:
62 12 Abhishek Lekshmanan
** if the suite was scheduled with *--machine-type plana,burnupi,mira*:
63 10 Abhishek Lekshmanan
<pre>
64 10 Abhishek Lekshmanan
./virtualenv/bin/teuthology-kill -m multi -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer 
65 10 Abhishek Lekshmanan
</pre>
66 12 Abhishek Lekshmanan
** if the suite was scheduled with *--machine-type vps*
67 10 Abhishek Lekshmanan
<pre>
68 12 Abhishek Lekshmanan
./virtualenv/bin/teuthology-kill -m vps -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer
69 10 Abhishek Lekshmanan
</pre>
70 10 Abhishek Lekshmanan
71 1 Loïc Dachary
* if the run already started the *-m* option is not necessary
72 10 Abhishek Lekshmanan
<pre>
73 12 Abhishek Lekshmanan
./virtualenv/bin/teuthology-kill -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer
74 10 Abhishek Lekshmanan
</pre>
75 3 Loïc Dachary
76 1 Loïc Dachary
h3. Integration suites
77 1 Loïc Dachary
78 1 Loïc Dachary
Expected to be successfully run on the integration branch before "asking approval to the leads":http://ceph.com/docs/master/dev/development-workflow/#resolving-bug-reports-and-implementing-features (hence before asking QE to tests further)
79 1 Loïc Dachary
80 2 Loïc Dachary
"rados":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados
81 2 Loïc Dachary
"rgw":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rgw
82 2 Loïc Dachary
"rbd":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rbd
83 2 Loïc Dachary
"fs":https://github.com/ceph/ceph-qa-suite/tree/master/suites/fs
84 1 Loïc Dachary
85 1 Loïc Dachary
h3. QE suites
86 1 Loïc Dachary
87 1 Loïc Dachary
Expected to be successfully run on the release branch before "passing it to the person publishing the release":http://ceph.com/docs/master/dev/development-workflow/#cutting-a-new-stable-release.
88 1 Loïc Dachary
89 1 Loïc Dachary
h4. dumpling
90 1 Loïc Dachary
91 1 Loïc Dachary
rados
92 1 Loïc Dachary
rbd
93 1 Loïc Dachary
rgw
94 1 Loïc Dachary
fs
95 1 Loïc Dachary
ceph-deploy
96 1 Loïc Dachary
upgrade/dumpling
97 1 Loïc Dachary
98 1 Loïc Dachary
h4. firefly
99 1 Loïc Dachary
100 1 Loïc Dachary
rados
101 1 Loïc Dachary
rbd
102 1 Loïc Dachary
rgw
103 1 Loïc Dachary
fs
104 1 Loïc Dachary
krbd
105 1 Loïc Dachary
kcephfs
106 1 Loïc Dachary
samba
107 1 Loïc Dachary
ceph-deploy
108 1 Loïc Dachary
upgrade/firefly
109 1 Loïc Dachary
upgrade/dumpling-firefly-x (to giant)
110 1 Loïc Dachary
powercycle 
111 1 Loïc Dachary
112 1 Loïc Dachary
h4. giant
113 1 Loïc Dachary
114 1 Loïc Dachary
rados
115 1 Loïc Dachary
rbd
116 1 Loïc Dachary
rgw
117 1 Loïc Dachary
fs
118 1 Loïc Dachary
krbd
119 1 Loïc Dachary
kcephfs
120 1 Loïc Dachary
knfs
121 1 Loïc Dachary
haddop
122 1 Loïc Dachary
samba
123 1 Loïc Dachary
rest
124 1 Loïc Dachary
multimds
125 1 Loïc Dachary
multi-version
126 1 Loïc Dachary
upgrade/giant
127 1 Loïc Dachary
powecycle