Automated tests AKA nightlies

The sepia lab runs automated teuthology tests on a regular basis. They are sometime called the nightlies. The schedule of the nightlies is managed by the QE team:

The following script is used in the above crontabs ( to schedule rados suite.

List of suites and watchers

Suite Watchers
smoke Sage, Yuri
powercycle Neha
rados Neha, Josh
rbd Jason
krbd Ilya
rgw Casey, Yehuda
fs Patrick, Venky
kcephfs Patrick, Venky
multimds Patrick, Venky
ceph-deploy Vasu
upgrade/*** Yuri
ceph-ansible Brad
teuthology ???
  • suites scheduled to run on current set of supported branches: luminous, mimic and master; master runs are used primarily for development purposes, upgrade* tests are not scheduled to run on master.


  • See HOWTO forensic analysis of integration and upgrade tests
  • sorting out the environmental noise
  • create new bugs for errors for which there is no match in the tracker
  • add a link to the failed job in pre-existing issues found in the tracker (useful to figure out the frequency and helps with debug when there are multiple outputs / logs)

The ceph-qa mailing list

The results of the nightlies are published for analysis on the ceph-qa mailing list. For a given release, the issues that make a job fail in the nightlies need to be backported with a higher priority to reduce the noise.