HOWTO describe a test result » History » Revision 5
Revision 4 (Loïc Dachary, 05/09/2015 09:33 AM) → Revision 5/8 (Nathan Cutler, 05/30/2015 02:19 PM)
h3. Format of teuthology analysis entries For instance http://tracker.ceph.com/issues/11153#rgw is structured like so: * In chronological order * The command line (that can be copy/pasted) used to run the suite * A bullet point with the URL to the suite run in pulpito prefixed by ** *running* if the run is not complete ** *red* if the run has at least one error ** *green* if the run has no error * If the run has at least one error, the output of the fail formatter snippet is added and edited when the errors are analyzed (call with *python fail.py loic-2015-04-21_10:20:06-rados-firefly-backports---basic-multi dead* or *python fail.py loic-2015-04-21_10:20:06-rados-firefly-backports---basic-multi fail*) <pre> import re import sys import requests paddle = ("http://paddles.front.sepia.ceph.com/runs/" + sys.argv[1] + "/jobs/?status=" + sys.argv[2]) pulpito = ("http://pulpito.ceph.com/" + sys.argv[1]) failure2jobs = {} def normalize(failure): if 'wget -O- ' in failure: return re.findall('"(.*)"', failure)[0] if 'Command failed' in failure: return re.findall('Command failed.*?:\s*(.*)', failure, )[0] else: return failure for job in requests.get(paddle).json(): failure2jobs.setdefault(normalize(job['failure_reason']), []).append(job) for (failure, jobs) in failure2jobs.iteritems(): print "** *" + failure + "*" for job in jobs: print '*** "' + job['description'] + '":' + pulpito + '/' + job['job_id'] </pre> The idea is to run the script after the entire suite completes (though this is not strictly necessary - jobs can be re-started even while the original suite is still running, but then one can easily get confused). The output of the script is copy/pasted into the release tracker issue. Then, each failure is analyzed and marked as belonging to one of the following categories: * When an error is analyzed, the link to the error is prefixed with ** *environmental noise* if it must be run again because it failed for reasons unrelated to the test itself (DNS error etc.) ** *known bug* and a URL to the bug (not just the number of the bug) ** *new bug* and a URL to the newly created bug if it was discovered during the analysis of this error: it is likely to be a regression ** *can be ignored* and a URL to the bug (not just the number of the bug) and the reason why it can be ignored (possibly a link to a mail thread)