Ceph » Stable releases

Automated tests AKA nightlies¶

The sepia lab runs automated teuthology tests on a regular basis. They are sometime called the nightlies. The schedule of the nightlies is managed by the QE team:

• Sepia
• Ovh

The following script is used in the above crontabs to schedule rados suite.

#!/bin/bash

# $1 - part
# $2 - branch name
# $3 - machine name
# $4 - email address
# $5 - filter out (this arg is to be at the end of the command line for now)

## example #1 
## (date +%U) week number
## % 2 - mod 2 (e.g. 0,1,0,1 ...)
## * 7 -  multiplied by 7 (e.g. 0,7,0,7...)
## $1 day of the week (0-6)
## /14 for 2 weeks

## example #2 
## (date +%U) week number
## % 4 - mod 4 (e.g. 0,1,2,3,0,1,2,3 ...)
## * 7 -  multiplied by 7 (e.g. 0,7,14,21,0,7,14,21...)
## $1 day of the week (0-6)
## /28 for 4 weeks

teuthology-suite -v -c $2 -m $3 -k distro -s rados --subset $(echo "(($(date +%U) % 4) * 7) + $1" | bc)/28 -e $4 $5

List of suites and watchers¶

• suites scheduled to run on current set of supported branches: hammer, infernalis, jewel and master; master runs are used primarily for development purposes, upgrade* tests are not scheduled to run on master.
• big and powercycle suites are run manually

Analyze¶

• See HOWTO forensic analysis of integration and upgrade tests
• sorting out the environmental noise
• create new bugs for errors for which there is no match in the tracker
• add a link to the failed job in pre-existing issues found in the tracker (useful to figure out the frequency and helps with debug when there are multiple outputs / logs)

The ceph-qa mailing list¶

The results of the nightlies are published for analysis on the ceph-qa mailing list. For a given release, the issues that make a job fail in the nightlies need to be backported with a higher priority to reduce the noise.