Project

General

Profile

Automated tests AKA nightlies

The sepia lab runs automated teuthology tests on a regular basis. They are sometime called the nightlies. The schedule of the nightlies is managed by the QE team:

The following script is used in the above crontabs to schedule rados suite.
https://github.com/ceph/ceph-qa-suite/blob/master/machine_types/schedule_rados.sh

List of suites and watchers

Suite Watchers
smoke Sage, Yuri
powercycle Yuri
rados Sage, Josh
rbd Jason
krbd Jason, Ilya
rgw Casey, Yehuda
fs Patrick
kcephfs Patrick, Zheng
knfs Patrick, Zheng
multimds Patrick
samba Patrick
ceph-deploy Alfredo, Yuri
upgrade/*** Yuri
ceph-ansible Vasu, Sebastian, Andrew, Alfredo
teuthology ???
  • suites scheduled to run on current set of supported branches: hammer, infernalis, jewel and master; master runs are used primarily for development purposes, upgrade* tests are not scheduled to run on master.
  • big and powercycle suites are run manually

Analyze

  • See HOWTO forensic analysis of integration and upgrade tests
  • sorting out the environmental noise
  • create new bugs for errors for which there is no match in the tracker
  • add a link to the failed job in pre-existing issues found in the tracker (useful to figure out the frequency and helps with debug when there are multiple outputs / logs)

The ceph-qa mailing list

The results of the nightlies are published for analysis on the ceph-qa mailing list. For a given release, the issues that make a job fail in the nightlies need to be backported with a higher priority to reduce the noise.