Tools » teuthology

alternatively, this could be done at teuthology-schedule level (not in teh sutie itself):

teuthology-schedule ... [--max NNN] [--sample .2]

where it would do the full expansion and then sample accordingly. e.g. if we expand to 500 tasks and --max is 15, it will pick 15 random ones out of the entire expansion. this would let us run jobs through the 'full suite' in sort of a '-lite' mode.

Just want to add what I said in stand-up: if we can achieve similar results by dividing tests into tiers, I'd prefer that as it's an industry standard method of dealing with testing bandwidth issues. That, and it wouldn't require adding complexity to an already-complex teuthology scheduler.

It would probably be best to implement this as a "deck of cards" kind of randomness. That is, if 20% of the tests were run, the next set of runs would be taken from the rest of the runs, thus guaranteeing that all tests are run. We would have to keep track of the seed and how far into the deck we are.

Also, potentially I can see problems with efficient utilization of resources. It may work out that one night, several time-consuming tests are run and another, things complete early. A possible improvement may be to start the tests, run at least the percentage of tests specified, and if time permits run more.

This scheduling could involve high priority runs that are always run. Then again, if one cares about specific test, they can be scheduled independently of these tests.

Just to add my 2c, adding a simple counter as an optional param to allow run 'n' number of first configurations will be very handy in addition to sampling.

is this obsoleted by --subset ?

Sage, Zack, Yuri, is this obsolete?