Project

General

Profile

Feature #23208

"add a feature to teuthology to be able to specify mix of OS/versions to be run"

Added by Yuri Weinstein almost 5 years ago. Updated almost 5 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Reviewed:
Affected Versions:

Description

Here is a chat from #sepia IRC for memory of what we discussed, we need to flash out details.

(12:21:29 PM) yuriw: I just realized that ever since we started using fog in sepia we have been under-testing other distros, b/c most of our suites don't ask for specific distro and we used to have percentages of diff OS/versions imaged on bare-metals before fog. So the questions now is what do we do ? schedule suites with "--distro XX --distro-version YY" - this will significantly increase amount of jobs or add add OS/versions to yaml configs with the same effect. Other option is to add a feature to teuthology to be able to specify mix of OS/versions to be run... this may allow us to use resources more retionally. well I'm just thinking aloud... Any takers for other options?
(12:21:52 PM) yuriw: sage joshd zackc dgalloway
(12:24:23 PM) yuriw: rationally
(01:20:41 PM) yuriw: joshd1 ^
(01:53:32 PM) zackc: yuriw: last time we talked about this, i suggested adding distros to suites where they're missing
(01:54:19 PM) dmick: perhaps pseudorandomly sampled
(01:54:58 PM) yuriw: zackc: I am not sure its practical, say you have 300+ rados jobs, adding distros will make them x4 !!
(01:55:14 PM) yuriw: what's "pseudorandomly sampled" ?
(01:56:29 PM) zackc: how would adding distros multiply?
(01:56:49 PM) zackc: i mean, sure if you add *multiple
distros i could see that
(01:58:05 PM) yuriw: we had a pool of smithi divided by dostro 30% each , adding distro will make load x time number of ditros
(01:58:14 PM) yuriw: distro*
(01:58:35 PM) gregsfortytwo: well we certainly want to run everything against more than one distro
(01:58:59 PM) dmick: that's what I meant, yeah; assuming you want to run the same amount of testing, but on a mix rather than a default distro
(01:59:38 PM) gregsfortytwo: I mean, if we add it as another dimension on the yaml fragment matrix that would get the mix
(01:59:55 PM) gregsfortytwo: be pretty annoying to just double it again though
(02:00:04 PM) yuriw: exactly ^
(02:00:33 PM) yuriw: or a new option in t-suite
(02:00:36 PM) joshd1: what's the issue with adding a separate facet for distro, and multiplying all the --subset denominators by number of distros?
(02:00:59 PM) yuriw: example ?
(02:02:12 PM) joshd1: e.g. if we have x/300 now, add a distro fragment with 4 distros to the suite, and run x/1200 instead
(02:03:00 PM) gregsfortytwo: means mucking around to make the crontab even more ridiculous to get the jobs down, and everybody who scheduled stuff manually has to adjust
(02:03:16 PM) yuriw: in other words, we use distro matrix in all suite (as oppose to none now) and use subset for all suites during scheduling
(02:03:38 PM) gregsfortytwo: do we have the ability to tell teuthology to fix a particular facet right now?
(02:03:57 PM) joshd1: sure, remove the other yaml files
(02:03:58 PM) yuriw: we could use --filter I'd say yes
(02:04:05 PM) vasu: yuriw: just add —distro and —distro version in cronjob and rotate it based on supported distros, like 1,3,5 for centos/ 2,6 for ubuntu
(02:04:29 PM) vasu: one thing we want is run smoke for all distros which will catch package related issues
(02:04:34 PM) vasu: and it runs fast
(02:05:08 PM) vasu: for others we can just run subsets and rotate based on the days its runs
(02:05:43 PM) yuriw: as ^ "means mucking around to make the crontab even more ridiculous"
(02:06:25 PM) vasu: nothing ridiculous i see, only kernel jobs are highly dependent on distros
(02:06:40 PM) vasu: running smoke on all distro's is more than sufficient to find package issues on daily basis
(02:07:24 PM) yuriw: smoke is a small and fast suite, nothing to compare vs rados and rbd
(02:08:20 PM) zackc: one feature i could see maybe being helpful here, is if we had another type of facet that would, instead of producing jobs containing each combination, would create only one job but with a random choice
(02:08:26 PM) zackc: not sure how helpful that would be, though
(02:08:51 PM) gregsfortytwo: yep, that's pretty much what we had before and would be nice
(02:09:04 PM) yuriw: yes but why random choice ?
(02:09:18 PM) yuriw: instead of desired or fixed
(02:10:05 PM) yuriw: say you asked for 3 distro, and you will get 1/3 off total amount of jobs ?
(02:11:09 PM) dmick: the idea would be "you want 1 job, but want it occasionally run on each of 3 distros"
(02:11:47 PM) vasu: zackc: are you saying add random choice instead of default ubuntu all the time?
(02:12:07 PM) yuriw: "you want 1 job" is applicable if you do manual scheduling, I am more worried about scheduled
(02:12:34 PM) dmick: of course I meant for a particular job, and no, I mean scheduled
(02:12:40 PM) dmick: never mind. others will explain better.
(02:12:53 PM) zackc: to explain my idea a bit more:
(02:13:14 PM) zackc: a % inside a directory tells us: "create one job using each of the facets in here"
(02:13:26 PM) zackc: so 3 distros => 3x the jobs
(02:14:27 PM) zackc: the idea would be to perhaps use a, i dunno, $, to tell us "create one job, selecting a random facet in here"
(02:14:47 PM) vasu: +1 for that ^

(02:15:16 PM) zackc: this would let you get some sort of distro coverage without increasing job load, and also without changing default behavior
(02:16:06 PM) vasu: yes makes sense
(02:17:47 PM) yuriw: well, maybe its Friday, but I did not get how it won't increase the total of jobs
(02:17:49 PM) yuriw: :(
(02:21:47 PM) dmick: "choose one of three vs choose three"
(02:22:54 PM) yuriw: if the result is always the same then I see
(02:45:20 PM) gregsfortytwo: no, it will select randomly on each invocation of the scheduler
(02:45:29 PM) gregsfortytwo: the point is to not always be the same
(02:45:39 PM) gregsfortytwo: but also not multiply the number of suites in a default run
(02:45:59 PM) gregsfortytwo: or require affirmative action from whoever's doing the scheduling to make sure we get a cycle
(02:56:12 PM) zackc: if you want the same thing every time, you have that right now

History

#1 Updated by Yuri Weinstein almost 5 years ago

  • Description updated (diff)

#2 Updated by Nathan Cutler almost 5 years ago

I, too, like Zack's idea:

(02:13:14 PM) zackc: a % inside a directory tells us: "create one job using each of the facets in here"
(02:13:26 PM) zackc: so 3 distros => 3x the jobs
(02:14:27 PM) zackc: the idea would be to perhaps use a, i dunno, $, to tell us "create one job, selecting a random facet in here"

#3 Updated by Sage Weil almost 5 years ago

Nathan Cutler wrote:

I, too, like Zack's idea:

(02:13:14 PM) zackc: a % inside a directory tells us: "create one job using each of the facets in here"
(02:13:26 PM) zackc: so 3 distros => 3x the jobs
(02:14:27 PM) zackc: the idea would be to perhaps use a, i dunno, $, to tell us "create one job, selecting a random facet in here"

I like that idea!

#4 Updated by Warren Usui almost 5 years ago

  • Assignee set to Warren Usui

I'll take a look at doing this.

#5 Updated by Warren Usui almost 5 years ago

Yuri. Could you try this with wip branch wusui-23208? Thanks.

Also available in: Atom PDF