Feature #9924
openAdd ability to expose test steps to pulpito
0%
Description
Currently we have no easy way to understand what steps a particular test case does until we look through yaml files code and do a memory refresh or reconstruct steps from there.
The current description looks like this:
description: upgrade:dumpling-firefly-x:stress-split/{00-cluster/start.yaml 01-dumpling-install/dumpling.yaml
02-partial-upgrade-firefly/firsthalf.yaml 03-workload/rbd.yaml 04-mona-upgrade-firefly/mona.yaml
05-workload/{rbd-cls.yaml readwrite.yaml} 06-monb-upgrade-firefly/monb.yaml 07-workload/{radosbench.yaml
rbd_api.yaml} 08-monc-upgrade-firefly/monc.yaml 09-workload/rbd-python.yaml 10-osds-upgrade-firefly/secondhalf.yaml
11-workload/snaps-few-objects.yaml 12-partial-upgrade-x/first.yaml 13-workload/rados_loadgen_big.yaml
14-mona-upgrade-x/mona.yaml 15-workload/rbd-import-export.yaml 16-monb-upgrade-x/monb.yaml
17-workload/readwrite.yaml 18-monc-upgrade-x/monc.yaml 19-workload/radosbench.yaml
20-osds-upgrade-x/osds_secondhalf.yaml 21-final-workload/{rados_stress_watch.yaml
rbd_cls_tests.yaml rgw-swift.yaml} distros/centos_6.5.yaml}
One way to address this and add English words to tests would be to enable test designers to add steps (open text comments) into yaml files and later expose those in pulpito as test results.
Say if we had a simple test, it could look like:
0-cluster/start.yaml
{steps}: "Create a cluster such and such..."
1-dumpling-install/dumpling.yaml
{steps}: "Install ..."
2-upgrade-sequence/upgrade-all.yaml
{steps}" "Restart osd and then ..."
By combining and displaying the list of {steps} in puplito it may serve as self-documenting feature.
This can also serve as a base for test plans for future releases.
Updated by Loïc Dachary over 9 years ago
When I read teuthology.log such as
2014-10-27T09:13:57.589 INFO:teuthology.task.install:Removing packages: ceph, ceph-dbg, ceph-mds, ceph-mds-dbg, ceph-common, ceph-common-dbg, ceph-fuse, ceph-fuse-dbg, ceph-test, ceph-test-dbg, radosgw, radosgw-dbg, python-ceph, libcephfs1, libcephfs1-dbg, libcephfs-java on Debian system.
it is straightforward to relate this to the install task. However when I read
2014-10-27T09:13:57.354 INFO:teuthology.orchestra.run.plana63:Running: "sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq"
The teuthology.orchestra.run part could be replaced with ceph-qa-suite/tasks/xxx.py otherwise it is necessary to go back in the history to figure out what tasks has been started on this machine. I guess the same applies to workunits / exec.
Updated by Ian Colle over 9 years ago
- Subject changed from Add ability to expose tests steps to pulpito to Add ability to expose test steps to pulpito
Updated by Vasu Kulkarni about 9 years ago
Someone has to write the steps in yaml/suite file then i think it can be propogated all they way upto pulpito, it should be easier for upper reporting structure using model like job.steps
Updated by Yuri Weinstein about 9 years ago
My 2c things on a wish list:
- ability to annotate/note yaml tests
- display notes via pulpito results
- ability to output notes (maybe via optional --test_steps arg) as test plan either/or in a file or standard out (this can be used in the future if we decide to publish test plans somewhere via HTML files)
Updated by Loïc Dachary about 9 years ago
It would be great if the description was displayed instead of the job number, date etc. when looking at details such as http://pulpito.ceph.com/teuthology-2015-04-01_23:20:01-upgrade:client-upgrade-master-distro-basic-multi/. I don't think I've ever used any of these fields when analyzing a test result. Each file in the description could be a URL to the corresponding file at https://ceph.com/git/?p=ceph-qa-suite.git so that it can be browsed to understand what it does, if needed.