HOWTO describe a test result » History » Version 8
Loïc Dachary, 02/26/2016 06:03 AM
1 | 1 | Loïc Dachary | h3. Format of teuthology analysis entries |
---|---|---|---|
2 | |||
3 | 8 | Loïc Dachary | For instance http://tracker.ceph.com/issues/14692#note-5 is structured like so: |
4 | 1 | Loïc Dachary | |
5 | * In chronological order |
||
6 | * The command line (that can be copy/pasted) used to run the suite |
||
7 | * A bullet point with the URL to the suite run in pulpito prefixed by |
||
8 | ** *running* if the run is not complete |
||
9 | 7 | Loïc Dachary | ** *fail* if the run has at least one error |
10 | 1 | Loïc Dachary | ** *green* if the run has no error |
11 | 6 | Loïc Dachary | * If the run has at least one error, the output of the fail formatter snippet is added and edited when the errors are analyzed (call with *python fail.py loic-2015-04-21_10:20:06-rados-firefly-backports---basic-multi fail* or *python fail.py loic-2015-04-21_10:20:06-rados-firefly-backports---basic-multi fail 167.114.249.14* if using teuthology-openstack and the IP is where the pulpito results are available) |
12 | 2 | Loïc Dachary | <pre> |
13 | import re |
||
14 | import sys |
||
15 | import requests |
||
16 | 1 | Loïc Dachary | |
17 | 6 | Loïc Dachary | if len(sys.argv) > 3: |
18 | host = sys.argv[3] |
||
19 | paddle_host = host + ":8080" |
||
20 | pulpito_host = host + ":8081" |
||
21 | else: |
||
22 | paddle_host = 'paddles.front.sepia.ceph.com' |
||
23 | pulpito_host = 'pulpito.ceph.com' |
||
24 | |||
25 | paddle = ("http://" + paddle_host + "/runs/" + |
||
26 | 2 | Loïc Dachary | sys.argv[1] + |
27 | 3 | Loïc Dachary | "/jobs/?status=" + sys.argv[2]) |
28 | 6 | Loïc Dachary | pulpito = ("http://" + pulpito_host + "/" + |
29 | 2 | Loïc Dachary | sys.argv[1]) |
30 | failure2jobs = {} |
||
31 | |||
32 | def normalize(failure): |
||
33 | if 'wget -O- ' in failure: |
||
34 | return re.findall('"(.*)"', failure)[0] |
||
35 | if 'Command failed' in failure: |
||
36 | return re.findall('Command failed.*?:\s*(.*)', failure, )[0] |
||
37 | else: |
||
38 | return failure |
||
39 | |||
40 | |||
41 | for job in requests.get(paddle).json(): |
||
42 | failure2jobs.setdefault(normalize(job['failure_reason']), []).append(job) |
||
43 | |||
44 | for (failure, jobs) in failure2jobs.iteritems(): |
||
45 | print "** *" + failure + "*" |
||
46 | 1 | Loïc Dachary | for job in jobs: |
47 | 2 | Loïc Dachary | print '*** "' + job['description'] + '":' + pulpito + '/' + job['job_id'] |
48 | 3 | Loïc Dachary | |
49 | 2 | Loïc Dachary | </pre> |
50 | 5 | Nathan Cutler | |
51 | The idea is to run the script after the entire suite completes (though this is not strictly necessary - jobs can be re-started even while the original suite is still running, but then one can easily get confused). The output of the script is copy/pasted into the release tracker issue. Then, each failure is analyzed and marked as belonging to one of the following categories: |
||
52 | |||
53 | 1 | Loïc Dachary | * When an error is analyzed, the link to the error is prefixed with |
54 | ** *environmental noise* if it must be run again because it failed for reasons unrelated to the test itself (DNS error etc.) |
||
55 | ** *known bug* and a URL to the bug (not just the number of the bug) |
||
56 | ** *new bug* and a URL to the newly created bug if it was discovered during the analysis of this error: it is likely to be a regression |
||
57 | 4 | Loïc Dachary | ** *can be ignored* and a URL to the bug (not just the number of the bug) and the reason why it can be ignored (possibly a link to a mail thread) |