Tools » teuthology

It makes sense for the teuthology program itself to be more independent of all the tasks. I am not sure what the right way of doing this is, but someone in the future may not want to have teuthology but not have all the excess tasks that they are not interested in testing. Also, we may have customers using teuthology who want to develop their own tasks but not make them publically available.

1. crontab -> run-suites.sh (simple wrapper around update.sh and schedule_suite.sh)
2. run-suites.sh -> update.sh (checks out ceph-qa-suite branch)
3. run-suites.sh -> schedule_suite.sh (decides on teuthology branch)
4. schedule_suite.sh -> teuthology-suite (generates job configs)
5. teuthology-suite -> teuthology-schedule
6. teuthology-schedule -> beanstalkd
7. beanstalkd -> teuthology-worker (checks out teuthology branch)
8. teuthology-worker -> teuthology (actually runs job)

1. Merge teuthology tasks with ceph-qa-suite
2. Create a teuthology-tasks repo and add it as a submodule to ceph-qa-suite
3. Create a teuthology-tasks repo and add it as a submodule to teuthology
4. Create a teuthology-tasks repo and keep it fully separate

1. Could definitely work; would potentially have to re-work update.sh.
2. Could work with similar caveats to the 'merge' option, but would be more complex
3. Would add too much complexity IMO
4. Could also work well.

I think I'm down to 'merge' or 'separate'; I'll come up with a more detailed analysis before moving forward.

Zack Cerza wrote:

Here are the possibilities that I see for how to organize the new teuthology, tasks, and ceph-qa-suite:

1. Merge teuthology tasks with ceph-qa-suite
2. Create a teuthology-tasks repo and add it as a submodule to ceph-qa-suite
3. Create a teuthology-tasks repo and add it as a submodule to teuthology
4. Create a teuthology-tasks repo and keep it fully separate

And here's what I think about them:

1. Could definitely work; would potentially have to re-work update.sh.
2. Could work with similar caveats to the 'merge' option, but would be more complex
3. Would add too much complexity IMO
4. Could also work well.

I think I'm down to 'merge' or 'separate'; I'll come up with a more detailed analysis before moving forward.

I'm inclined toward #1 just because it means a single repo for the tests, which are frequently updated together. Also simpler, fewer moving parts, etc.

update.sh is a hack so not too worried about that :)

Yeah, I'd been leaning toward that one.

I'm thinking it will make sense to roll run-suites.sh and update.sh into schedule_suite.sh - but I also want all that to be written in python so that it can share code with the rest of teuthology.

Edit: #6343

Now that #6343 is done, here is what needs to happen:

• tasks need to be split from teuthology and added to ceph-qa-suite in a tasks/ subdir
• teuthology-suite needs to record which suite_branch is used in job configs
• teuthology-worker needs to start checking out copies of ceph-qa-suite - the branch is either the same as the ceph branch, or the suite_branch passed to teuthology-suite
• I need to figure out how teuthology should import the tasks. Perhaps adding .../ceph-qa-suite_{branch}/tasks/ to sys.path in teuthology.run_tasks.

The teuthology-suite and teuthology-worker changes will probably happen in master before anything else. If that's too scary then I can create a separate branch, and have the scheduler and workers use that branch - but that's more complicated and not necessarily more helpful.

Once that is done, I think making the run_tasks changes might come next - so that a few jobs can be run to shake out any bugs before the actual split happens. That way, I will also be able to test the split separately in dedicated teuthology and ceph-qa-suite branches (let's say split_tasks and merge_tasks respectively) and fix the inevitable bugs. Then, we can merge those in to the respective master branches and make the corresponding changes to teuthology's and ceph-qa-suite's dumpling and firefly branches.

I'm doing preliminary work in this branch:
https://github.com/ceph/teuthology/compare/split_prep?expand=1

In my typical fashion, the branch contains some refactoring, enhancements to existing chunks of the framework, and unit tests for those changes. I plan to use those changes in other areas of teuthology when I touch them next.

• teuthology-suite recording the suite_branch and storing it in the job config
• teuthology-worker using suite_branch to check out a copy of ceph-qa-suite
• teuthology-worker adding the ceph-qa-suite checkout to the PYTHONPATH of the job so it picks up the right tasks

• Get that branch reviewed and merged, and shake out bugs
• Split tasks in a teuthology branch and add them to a ceph-qa-suite branch
• Run a few jobs to shake out bugs
• Fix the bugs
• Backport the splits to the dumpling and firefly branches of teuthology and ceph-qa-suite

Zack Cerza wrote:

• teuthology-worker using suite_branch to check out a copy of ceph-qa-suite

I had to remove this since it relied on GitPython, and it turns out it's buggy and unmaintained.

PR: https://github.com/ceph/teuthology/pull/289

Now that it seems the pre-split work is done and the bugs are all fixed, I'm working on the actual logistics of splitting out teuthology.git/teuthology/task/, retaining its history, and moving into ceph-qa-suite.git/tasks/.

Alright. I have the above figured out: a script that can import tasks from teuthology.git to ceph-qa-suite.git while retaining each file's history. It will be easy to make it also remove the tasks from teuthology.git.

• teuthology/task_util/ - move it along with the tasks? Maybe tasks/util/?
• Adjusting all the references that cross the teuthology <-> /tasks/ boundary.

Lots of tasks do things like:

from ..orchestra import run

They'll have to be modified to do (I think):

from teuthology.orchestra import run

Also, teuthology.run_tasks does:

from .task import interactive

I think it will simply have to be:

from tasks import interactive

I think all these will have to be adjusted manually.

Maybe it's time to start that process with one of the branches. I might want to do dumpling first since master may have PRs and other changes come in while I'm doing all this. An outline of the process could look like:

1. Create a teuthology/split_dumpling based on teuthology/dumpling
2. Create a teuthology/split_master based on teuthology/master
3. Create a ceph-qa-suite/merge_dumpling based on ceph-qa-suite/dumpling
4. Move teuthology.task and teuthology.task_util from teuthology to ceph-qa-suite
5. Update references in /tasks/
6. Update teuthology/split_master to look for tasks in the right place
7. Port tasks in ceph-qa-suite/merge_dumpling to use updated APIs in teuthology/split_master
8. Run some tests using teuthology/split_master and ceph-qa-suite/merge_dumpling
9. Fix bugs
10. Merge ceph-qa-suite/merge_dumpling with ceph-qa-suite/dumpling
11. Delete teuthology/split_dumpling and teuthology/dumpling

Also, another problem to be solved is: how to handle the running of non-scheduled jobs. I suppose we'll have to use repo_utils to fetch the ceph-qa-suite repo; shouldn't be a big deal.

The migration script was almost done when I realized that teuthology actually uses code from a few of the tasks. Sage and I decided that the right thing to do would be to actually keep a couple of the tasks inside teuthology - so we'd have 'internal' tasks and 'external' tasks.

I'm going through everything right now to decide which will be which.

Note to self: Don't forget to update dumpling tasks to reflect the movement of adjust-ulimits to $PATH