Tools » teuthology

This is in the plans after the new sepia hardware is in place; current sepia re-install is too slow & painful to dare to use debs, and with debs we really need to reinstall machines all the time to have a known baseline.

Mark Kampe wrote:

This proposal does not eliminate /tmp/cephtest, but it should move us to more standard pacakge installation. I believe all of the changes are to the ceph.py task, mostly in the binaries method.

(1) support yet another form of "stuff to test", a deb or directory full of debs, which will be installed in the default place (rather than /tmp/cephtest/binaries).

(2) change the finally case to remove and then purge the packages.

(3) make sure that purging gets rid of the stuff in {/etc,/var/log,/var/run}/ceph, and that the stale machine tests look in all these places for evidence of an unclean system.

(4) The 'cluster' method currently creates configuration files and runs commands based on a hard-coded '/tmp/cephtest/binary' and this would have to become a variable set by in the 'binaries' method ... or perhaps simply another yaml specifiable parameter.

Until reimageing is working, this cannot be made the default for teuthology runs so the target directory still needs to default to /tmp/cephtest/binary; This seems unnatural, but we can reopen the question of default values later.

Is that actually true? I don't think dpkg --purge with /etc/ceph etc. checks is any worse than the /tmp/cephtest install we're doing now.

Note, however, that all of the other files we create as part of a test might just as well continue being placed in the /tmp/cephtest directory.

We'll probably want to be able to remove any ceph related packages with teuthology-nuke as well.

First, teuthology actually started life as something that installed debs. It did not work well. I would highly recommend against trying to kludge teuthology-nuke into being able to handle all the failure cases right. To do debs, we should do reinstalls/reimages.

Second, we should use apt repositories to pull the debs from. Not testing the actual installation mechanism is just asking for bugs in that part.

Third, longer term, I'd really like to use the chef cookbooks to do the configuration needed, so that the product and the tests are actually more related to each other; currently teuthology setups and production setups look too different.

I'd like to handle the teuthology change as basically a refactoring of teuthology, discarding the old tarball style. It's just going to look too different to comfortably handle both ways. While the new stuff isn't reliable yet, developers & regression can run an older version of teuthology.

I tried to assign this to myself but I'm not sure I'm allowed.

You're not sure you're allowed, or you tried and it doesn't work? The answer is "yes", and "if so we'll fix it"

Just a heads up, I think there are quite a few tasks now that assume /tmp/cephtest exists. I've been assuming that in my task writing (sorry - I didn't know about this bug till just now!), and I've seen other places that assume the same.

Yes, I'm also concerned about that. I already made notes to make sure the start-up works correctly after the install but when it comes to tasks it's a whole different ball of wax.

Dan: I wrote "I'm not sure" because I've never used Redmine before and it's certainly possible I don't see how to do it which is different than stating I'm certain I'm not allowed. Call it sandbagging born from many years of doing QA.

Proposed steps for installing a specific set of debian packages:

- using existing logic to determine sha1 key use wget to obtain the build:
wget -r --no-parent --cut-dirs=9 http://gitbuilder.ceph.com/ceph-deb-natty-x86_64-basic/sha1/017b6d63db41473645f38589daceccf174972136/pool/main/c/ceph/
- remove *dbg* packages
- sudo dpkg -i *.deb
- sudo apt-get -f install
- modify the configuration code that assumes /tmp/ceph to use installed packages

To be honest I'm not familiar with using apt-get to pull specific builds from gitbuilder so this is what I came up with. My model for installing is to pull packages from the build repository and install those, similar to what was described above. Please suggest alternatives if there's a better way to do this.

I think you can use the apt sources list stuff to do this. http://www.ceph.com/docs/master/install/debian/#development-testing-packages has some basic instructions.

If we want clean room installs (the equivalent of rm -rf /tmp/cephtest), you can use overlayfs and chroot to install everything into a temp dir that's stacked ontop of your root dir. Something like this:

mkdir /tmp/teuth-run-abcd
mkdir -p /tmp/teuth-root
mount -t overlayfs -o lowerdir=/,upperdir=/tmp/teuth-run-abcd overlayfs /tmp/teuth-root
chroot /tmp/teuth-root /bin/bash

... install packages, run tasks, etc ...

The hard part about something like this is getting the /proc and /sys mounts setup correctly for your chroot. It should be doable though. This also solves the 'test did not cleanup properly' problem (#3767), as the state from previous teuthology runs that failed can be saved in /tmp/teuth-run-whatever.

I think either wget + dpkg -i on the debs, or apt-get would work. Either way, we need to remove --purge the packages before, so it shouldn't much matter. apt-get is maybe marginally easier to implement, since you just need to update the sources.d/ceph.list file and the apt-get install. OTOH, you need to run apt-get update, which is slower, so the wget + dpkg -i might be fastest to actually run.

I think /tmp/cephtest will stick around.. it just won't have /tmp/cephtest/binary with the build tarball.

Thanks for your comments. Good to know both are viable options. I'm now experimenting with overlayfs and chroot. After chroot /etc/resolv.conf is empty so apt-get fails. It may be easier to download packages first, then chroot and dpkg -i

Update: apt-get gets the right packages (ceph.list is also updated) but fails due to the non-interactive nature of the ssh connection:

INFO:teuthology.orchestra.run.out:Hit http://us.archive.ubuntu.com precise-backports/multiverse Translation-en
INFO:teuthology.orchestra.run.out:Hit http://us.archive.ubuntu.com precise-backports/restricted Translation-en
INFO:teuthology.orchestra.run.out:Hit http://us.archive.ubuntu.com precise-backports/universe Translation-en
INFO:teuthology.orchestra.run.out:Fetched 2,738 kB in 7s (364 kB/s)
INFO:teuthology.orchestra.run.out:The following packages were automatically installed and are no longer required:
INFO:teuthology.orchestra.run.out:  msr-tools seabios cpu-checker vgabios kvm-ipxe bridge-utils qemu-common
INFO:teuthology.orchestra.run.out:Use 'apt-get autoremove' to remove them.
INFO:teuthology.orchestra.run.out:The following packages will be upgraded:
INFO:teuthology.orchestra.run.out:  ceph
INFO:teuthology.orchestra.run.out:1 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.
INFO:teuthology.orchestra.run.out:Need to get 5,180 kB of archives.
INFO:teuthology.orchestra.run.out:After this operation, 403 kB of additional disk space will be used.
INFO:teuthology.orchestra.run.out:Get:1 http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/argonaut/ precise/main ceph amd64 0.48.3argonaut-3-g7e8e649-1precise [5,180 kB]
INFO:teuthology.orchestra.run.err:debconf: unable to initialize frontend: Dialog
INFO:teuthology.orchestra.run.err:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
INFO:teuthology.orchestra.run.err:debconf: falling back to frontend: Readline
INFO:teuthology.orchestra.run.err:debconf: unable to initialize frontend: Readline
INFO:teuthology.orchestra.run.err:debconf: (This frontend requires a controlling tty.)
INFO:teuthology.orchestra.run.err:debconf: falling back to frontend: Teletype
INFO:teuthology.orchestra.run.err:dpkg-preconfigure: unable to re-open stdin:
INFO:teuthology.orchestra.run.out:Fetched 5,180 kB in 0s (32.7 MB/s)
INFO:teuthology.orchestra.run.err:debconf: unable to initialize frontend: Dialog
INFO:teuthology.orchestra.run.err:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
INFO:teuthology.orchestra.run.err:debconf: falling back to frontend: Readline
INFO:teuthology.orchestra.run.err:debconf: unable to initialize frontend: Readline
INFO:teuthology.orchestra.run.err:debconf: (This frontend requires a controlling tty.)

There are plenty of reference to this problem but no solution yet.

Sander Pool wrote:

Update: apt-get gets the right packages (ceph.list is also updated) but fails due to the non-interactive nature of the ssh connection:
[...]

There are plenty of reference to this problem but no solution yet.

I think a combination of uninstalling the previous version, and installing with -y will work..

and/or -y --force-yes ?

Otherwise, it's a "=$version" suffix for each package listed, which is slightly annoying to deduce. IIRC there is a 'version' file in each build directory generated by gitbuilder that you can wget.. that'll have the git describe version, add "-1" to that for the debianization

-y and --force-yes works, thanks!