Tools » teuthology

We've got a lot of people trying to push a lot of full suite runs through the system. It would be great if we could automatically prune the test suite down to a smaller size, without sacrificing very much coverage. (I am presently doing this manually, by deleting fragments that are redundant for my testing — eg, two of the three local filesystems we test on.)
The simplest idea I've come up with is the ability to mark certain fragment collections as "pick one", and then simply alternating back and forth between them on subsequent branch runs or something. For filesystem tests we would set this property on the collection which determines if we use inline data or not, for instance.
I've heard Sam and Sage discuss more complicated ideas, like setting priorities on each fragment, deterministically ordering each possible test based on those priorities, and then running a given portion of the tests in order; or starting from a given test and running through n of them on each run, or whatever.

But we need some way to reduce both the size of our personal tests, and of the nightlies, in a way that is repeatable but keeps running every combination on a regular basis.