HOWTO monitor the automated tests AKA nightlies » History » Revision 27
Revision 26 (Loïc Dachary, 03/31/2016 11:30 AM) → Revision 27/38 (Yuri Weinstein, 04/21/2016 10:01 PM)
h3. Automated tests AKA nightlies The "sepia lab":http://ceph.com/docs/master/dev/sepia/ runs automated teuthology tests on a regular basis. They are sometime called the *nightlies*. The schedule of the nightlies is managed by the QE team: * [[Sepia]] * [[Ovh]] The following script is used in the above crontabs to schedule @rados@ suite. <pre> #!/bin/bash # $1 - part # $2 - branch name # $3 - machine name # $4 - email address # $5 - filter out (this arg is to be at the end of the command line for now) ## example #1 ## (date +%U) week number ## % 2 - mod 2 (e.g. 0,1,0,1 ...) ## * 7 - multiplied by 7 (e.g. 0,7,0,7...) ## $1 day of the week (0-6) ## /14 for 2 weeks ## example #2 ## (date +%U) week number ## % 4 - mod 4 (e.g. 0,1,2,3,0,1,2,3 ...) ## * 7 - multiplied by 7 (e.g. 0,7,14,21,0,7,14,21...) ## $1 day of the week (0-6) ## /28 for 4 weeks teuthology-suite -v -c $2 -m $3 -k distro -s rados --subset $(echo "(($(date +%U) % 4) * 7) + $1" | bc)/28 -e $4 $5 </pre> h3. List of suites and watchers | *Suite* | *Watchers* | | smoke | Sage, Yuri | | big | Sage, Sam | | powercycle | Yuri | | rados | Sage, Sam | | rbd | Josh | | krbd | Josh, Ilya | | rgw | Yehuda | | fs | Greg, John | | kcephfs | Greg, John, Zheng | | knfs | Greg, John, Zheng | | multimds | Greg, John, Zheng | | samba | Greg, John | | ceph-deploy | Alfredo, Travis, Yuri | | upgrade/*** upgrade/client-upgrade | Yuri | | ceph-ansible upgrade/dumpling-firefly-x | ??? Yuri | | teuthology upgrade/firefly-x | ??? Yuri | | upgrade/giant-x | Yuri | | upgrade/firefly | Yuri | | upgrade/hammer | Yuri | | upgrade/giant | Yuri | * suites scheduled to run on current set of supported branches: *hammer*, *infernalis*, *jewel* and *master*; *master* runs are used primarily for development purposes, *upgrade** tests are not scheduled to run on master. * *big* and *powercycle* suites are run manually h3. Analyze * See [[HOWTO forensic analysis of integration and upgrade tests]] * sorting out the environmental noise * create new bugs for errors for which there is no match in the tracker * add a link to the failed job in pre-existing issues found in the tracker (useful to figure out the frequency and helps with debug when there are multiple outputs / logs) h3. The ceph-qa mailing list The results of the nightlies are published for analysis on the "ceph-qa mailing list":http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com/. For a given release, the issues that make a job fail in the nightlies need to be backported with a higher priority to reduce the noise.