Ceph » Stable releases

h3. Automated tests AKA nightlies

The "sepia lab":http://ceph.com/docs/master/dev/sepia/ runs automated teuthology tests on a regular basis. They are sometime called the *nightlies*. The schedule of the nightlies is managed by the QE team:

* [[Sepia]]

* [[Ovh]]

The following script is used in the above crontabs to schedule @rados@ suite.

<pre>

#!/bin/bash

# $1 - part

# $2 - branch name

# $3 - machine name

# $4 - email address

# $5 - filter out (this arg is to be at the end of the command line for now)

## example #1 

## (date +%U) week number

## % 2 - mod 2 (e.g. 0,1,0,1 ...)

## * 7 -  multiplied by 7 (e.g. 0,7,0,7...)

## $1 day of the week (0-6)

## /14 for 2 weeks

## example #2 

## (date +%U) week number

## % 4 - mod 4 (e.g. 0,1,2,3,0,1,2,3 ...)

## * 7 -  multiplied by 7 (e.g. 0,7,14,21,0,7,14,21...)

## $1 day of the week (0-6)

## /28 for 4 weeks

teuthology-suite -v -c $2 -m $3 -k distro -s rados --subset $(echo "(($(date +%U) % 4) * 7) + $1" | bc)/28 -e $4 $5

</pre>

h3. List of suites and watchers

| *Suite*                    | *Watchers* |

| smoke                      | Sage, Yuri |

| big                        | Sage, Sam |

| powercycle                 | Yuri |

| rados                      | Sage, Sam |

| rbd                        | Josh |

| krbd                       | Josh, Ilya  |

| rgw                        | Yehuda |

| fs                         | Greg, John  |

| kcephfs                    | Greg, John, Zheng |

| knfs                       | Greg, John, Zheng  |

| multimds                   | Greg, John, Zheng  |

| samba                      | Greg, John  |

| ceph-deploy                | Travis, Yuri |

| upgrade/client-upgrade     | Yuri  |

| upgrade/dumpling-firefly-x | Yuri  |

| upgrade/firefly-x          | Yuri  |

| upgrade/giant-x            | Yuri  |

| upgrade/firefly            | Yuri  |

| upgrade/hammer             | Yuri  |

| upgrade/giant              | Yuri  |

*  suites scheduled to run on current set of supported branches: *hammer*, *infernalis*, *jewel* and *master*; *master* runs are used primarily for development purposes, *upgrade** tests are not scheduled to run on master.

*  *big* and *powercycle* suites are run manually

h3. Analyze

* See [[HOWTO forensic analysis of integration and upgrade tests]]

* sorting out the environmental noise

* create new bugs for errors for which there is no match in the tracker

* add a link to the failed job in pre-existing issues found in the tracker (useful to figure out the frequency and helps with debug when there are multiple outputs / logs)

h3. The ceph-qa mailing list

The results of the nightlies are published for analysis on the "ceph-qa mailing list":http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com/. For a given release, the issues that make a job fail in the nightlies need to be backported with a higher priority to reduce the noise.