HOWTO run integration and upgrade tests » History » Version 18
Loïc Dachary, 05/07/2015 07:36 AM
1 | 3 | Loïc Dachary | h3. Scheduling a suite |
---|---|---|---|
2 | 3 | Loïc Dachary | |
3 | 3 | Loïc Dachary | This requires access to a running teuthology cluster, for instance the cluster from the sepia lab. |
4 | 3 | Loïc Dachary | |
5 | 3 | Loïc Dachary | * git clone https://github.com/ceph/teuthology/ |
6 | 3 | Loïc Dachary | * cd teuthology |
7 | 3 | Loïc Dachary | * ./bootstrap |
8 | 8 | Loïc Dachary | * Test the suite with *--dry-run*, i.e. something like: |
9 | 3 | Loïc Dachary | <pre> |
10 | 15 | Loïc Dachary | ./virtualenv/bin/teuthology-suite --dry-run --filter-out btrfs,ext4 --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph giant |
11 | 1 | Loïc Dachary | </pre> |
12 | 6 | Loïc Dachary | * Review the jobs to be scheduled and if it matches what is expected, run the same command without the *--dry-run* |
13 | 9 | Loïc Dachary | * Assuming the suite was run on the sepia lab, it will immediately show up at http://pulpito.ceph.com/ (for instance http://pulpito.ceph.com/loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer---basic-multi/). Note the *loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer---basic-multi* part of the path: it matches the run name displayed by teuthology-suite |
14 | 3 | Loïc Dachary | |
15 | 6 | Loïc Dachary | The meaning of the teuthology-suite arguments are: |
16 | 4 | Loïc Dachary | |
17 | 4 | Loïc Dachary | * *--suite* a reference to https://github.com/ceph/ceph-qa-suite/tree/master/suites. For instance --suite rados means run all jobs at https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados |
18 | 4 | Loïc Dachary | * *--suite-branch* a reference to the ceph-qa-suite branch to use. For instance --suite rados --suite-branch giant means run all jobs at https://github.com/ceph/ceph-qa-suite/tree/giant/suites/rados instead of https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados |
19 | 4 | Loïc Dachary | * *--priority* is the the priority of the job in the queue (lower numbers are higher priority). By default it is 1000 and if nothing is urgent it should be used. If in the process of debugging a single job that fails, the priority 101 can be used so that it is scheduled faster. |
20 | 4 | Loïc Dachary | * *--machine-type* is the kind of machine that will be provisionned to run the tests. Use *plana,burnupi,mira* for bare metal or *vps* for virtual machines. |
21 | 4 | Loïc Dachary | * *--ceph* is the branch of the Ceph repository to use and defaults to master. For instance --ceph giant means use https://github.com/ceph/ceph/tree/giant/ instead of https://github.com/ceph/ceph/tree/master/ |
22 | 4 | Loïc Dachary | * *--email* when the run is complete an email will be sent to this address with a summary of the results. |
23 | 4 | Loïc Dachary | |
24 | 16 | Loïc Dachary | h3. Scheduling a rados suite |
25 | 13 | Loïc Dachary | |
26 | 18 | Loïc Dachary | *NOTE*: <code>git checkout 819c63a8fa9776028311b96efe2c14d97291ff8f</code> can be used until the code is merged. It was "reverted because of a bug":http://tracker.ceph.com/issues/11533 |
27 | 18 | Loïc Dachary | |
28 | 16 | Loïc Dachary | In hammer "rados suite":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados generates more than 3000 jobs and it is too heavy for the purpose of validating a series of pull requests before they are merged into the stable release branch. The number of jobs can be reduced by using the *--subset* option. It will make sure every yaml (facet) is included in the run but will not include all the combinations. For instance: |
29 | 13 | Loïc Dachary | |
30 | 16 | Loïc Dachary | <code>./virtualenv/bin/teuthology-suite --subset 1/7 --priority 101 --suite rados --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports</code> |
31 | 13 | Loïc Dachary | |
32 | 17 | Loïc Dachary | Will reduce the number of jobs to *1/7* of the total number of jobs, as long as it includes all yaml (facets). |
33 | 17 | Loïc Dachary | |
34 | 17 | Loïc Dachary | For a given "rados suite":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados *--subset 0/7* will always create the same jobs. By running *0/7*, then *1/7*, etc. up to *7/7* all jobs are generated. It is a good idea to not always pick the same subset to get various combinations and increase the odds of discovering a problem. |
35 | 13 | Loïc Dachary | |
36 | 3 | Loïc Dachary | h3. Re-scheduling failed or dead jobs from an existing suite |
37 | 3 | Loïc Dachary | |
38 | 3 | Loïc Dachary | * Ask https://github.com/ceph/paddles (the server in which suite runs are stored) about the *dead* jobs |
39 | 3 | Loïc Dachary | <pre> |
40 | 3 | Loïc Dachary | run=loic-2015-03-23_01:09:31-rados-giant---basic-multi |
41 | 3 | Loïc Dachary | eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=dead | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') |
42 | 3 | Loïc Dachary | </pre> |
43 | 5 | Loïc Dachary | or the *fail* jobs: |
44 | 3 | Loïc Dachary | <pre> |
45 | 3 | Loïc Dachary | run=loic-2015-03-23_01:09:31-rados-giant---basic-multi |
46 | 1 | Loïc Dachary | eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') |
47 | 3 | Loïc Dachary | </pre> |
48 | 3 | Loïc Dachary | * Re-run the suite using the same command line without *--filter-out* and with *--filter "$filter"* to only schedule the jobs described in the *fitler* variable |
49 | 3 | Loïc Dachary | <pre> |
50 | 15 | Loïc Dachary | ./virtualenv/bin/teuthology-suite --filter "$filter" --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph giant |
51 | 3 | Loïc Dachary | </pre> |
52 | 3 | Loïc Dachary | |
53 | 7 | Loïc Dachary | h3. Killing a suite |
54 | 7 | Loïc Dachary | |
55 | 7 | Loïc Dachary | * if the run is scheduled but did not start yet: |
56 | 12 | Abhishek Lekshmanan | ** if the suite was scheduled with *--machine-type plana,burnupi,mira*: |
57 | 10 | Abhishek Lekshmanan | <pre> |
58 | 10 | Abhishek Lekshmanan | ./virtualenv/bin/teuthology-kill -m multi -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer |
59 | 10 | Abhishek Lekshmanan | </pre> |
60 | 12 | Abhishek Lekshmanan | ** if the suite was scheduled with *--machine-type vps* |
61 | 10 | Abhishek Lekshmanan | <pre> |
62 | 12 | Abhishek Lekshmanan | ./virtualenv/bin/teuthology-kill -m vps -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer |
63 | 10 | Abhishek Lekshmanan | </pre> |
64 | 10 | Abhishek Lekshmanan | |
65 | 1 | Loïc Dachary | * if the run already started the *-m* option is not necessary |
66 | 10 | Abhishek Lekshmanan | <pre> |
67 | 12 | Abhishek Lekshmanan | ./virtualenv/bin/teuthology-kill -r loic-2015-03-27_09:57:09-upgrade:firefly-x:stress-split-erasure-code-hammer |
68 | 10 | Abhishek Lekshmanan | </pre> |
69 | 3 | Loïc Dachary | |
70 | 1 | Loïc Dachary | h3. Integration suites |
71 | 1 | Loïc Dachary | |
72 | 1 | Loïc Dachary | Expected to be successfully run on the integration branch before "asking approval to the leads":http://ceph.com/docs/master/dev/development-workflow/#resolving-bug-reports-and-implementing-features (hence before asking QE to tests further) |
73 | 1 | Loïc Dachary | |
74 | 2 | Loïc Dachary | "rados":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rados |
75 | 2 | Loïc Dachary | "rgw":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rgw |
76 | 2 | Loïc Dachary | "rbd":https://github.com/ceph/ceph-qa-suite/tree/master/suites/rbd |
77 | 2 | Loïc Dachary | "fs":https://github.com/ceph/ceph-qa-suite/tree/master/suites/fs |
78 | 1 | Loïc Dachary | |
79 | 1 | Loïc Dachary | h3. QE suites |
80 | 1 | Loïc Dachary | |
81 | 1 | Loïc Dachary | Expected to be successfully run on the release branch before "passing it to the person publishing the release":http://ceph.com/docs/master/dev/development-workflow/#cutting-a-new-stable-release. |
82 | 1 | Loïc Dachary | |
83 | 1 | Loïc Dachary | h4. dumpling |
84 | 1 | Loïc Dachary | |
85 | 1 | Loïc Dachary | rados |
86 | 1 | Loïc Dachary | rbd |
87 | 1 | Loïc Dachary | rgw |
88 | 1 | Loïc Dachary | fs |
89 | 1 | Loïc Dachary | ceph-deploy |
90 | 1 | Loïc Dachary | upgrade/dumpling |
91 | 1 | Loïc Dachary | |
92 | 1 | Loïc Dachary | h4. firefly |
93 | 1 | Loïc Dachary | |
94 | 1 | Loïc Dachary | rados |
95 | 1 | Loïc Dachary | rbd |
96 | 1 | Loïc Dachary | rgw |
97 | 1 | Loïc Dachary | fs |
98 | 1 | Loïc Dachary | krbd |
99 | 1 | Loïc Dachary | kcephfs |
100 | 1 | Loïc Dachary | samba |
101 | 1 | Loïc Dachary | ceph-deploy |
102 | 1 | Loïc Dachary | upgrade/firefly |
103 | 1 | Loïc Dachary | upgrade/dumpling-firefly-x (to giant) |
104 | 1 | Loïc Dachary | powercycle |
105 | 1 | Loïc Dachary | |
106 | 1 | Loïc Dachary | h4. giant |
107 | 1 | Loïc Dachary | |
108 | 1 | Loïc Dachary | rados |
109 | 1 | Loïc Dachary | rbd |
110 | 1 | Loïc Dachary | rgw |
111 | 1 | Loïc Dachary | fs |
112 | 1 | Loïc Dachary | krbd |
113 | 1 | Loïc Dachary | kcephfs |
114 | 1 | Loïc Dachary | knfs |
115 | 1 | Loïc Dachary | haddop |
116 | 1 | Loïc Dachary | samba |
117 | 1 | Loïc Dachary | rest |
118 | 1 | Loïc Dachary | multimds |
119 | 1 | Loïc Dachary | multi-version |
120 | 1 | Loïc Dachary | upgrade/giant |
121 | 1 | Loïc Dachary | powecycle |