Tools » teuthology

I really wish shaman would just point us there; it is the source of truth after all :-/

shaman can't do that because these are builds that are out of the world-view of builds/repos that shaman knows about. Shaman is just meant to deal with dev repositories only (or at least that was the scope of the tool).

Although with some effort it might be possible make shaman point to other repos, I don't think that would work for download.ceph.com since we can't install 12.2.4 if 12.2.5 exists. This is a constraint in how these repos are built and pushed for release.

Yeah, I understand. This seems like a valid request, but I'm not immediately sure how to implement it using the chacra API, since we really only have the tag to go on.

We already do things like:
git ls-remote git://git.ceph.com/ceph v12.2.5^{}

To get a SHA1 (cad919881333ac92274171586c827e01f554a70a in this case), but that's not quite enough it seems, as https://chacra.ceph.com/search/?ref=cad919881333ac92274171586c827e01f554a70a just gives us [].

Edit: Perhaps chacra could allow us to query /search/?sha1=foo ?

I was thinking the workflow could be:

1. teuthology git ls-remote's git.ceph.com
2. translates the tag (e.g., 12.2.5) to the sha1
3. if no builds are found in shaman,
4. proceed to checking download.ceph.com

The tricky part I'm sure will require a fair bit of coding is translating the tag to a directory path. For example going from tag v12.2.5 to http://download.ceph.com/debian-luminous/ and only installing 12.2.5 packages.

Right; we don't currently have a way to go from sha1 -> branch, so we aren't really able to locate the download.ceph.com repo.

What is we had a lookup table with hardcoded values for download.ceph.com, e.g.:

v12.2.2 --- sha1xxx
v12.2.5 --- sha1yyy
v10.0.10 --- sha1zzz
....

they won't change, will they ?

We already do map tag -> sha1 with git ls-remote; the issue is the branch that chacra queries require

Yuri Weinstein wrote:

What is we had a lookup table with hardcoded values for download.ceph.com, e.g.:

v12.2.2 --- sha1xxx
v12.2.5 --- sha1yyy
v10.0.10 --- sha1zzz
....

they won't change, will they ?

No, they won't change, but this would merely replace one manual process (putting the build of a particular SHA1 onto a special chacra node that doesn't delete the repo after a short period of time) with another one (adding entries to the lookup table).

To get a SHA1 (cad919881333ac92274171586c827e01f554a70a in this case), but that's not quite enough it seems, as https://chacra.ceph.com/search/?ref=cad919881333ac92274171586c827e01f554a70a just gives us [].

Are you sure there really is a build of cad919881333ac92274171586c827e01f554a70a in chacra? IIRC there is a special chacra node that doesn't autodelete its repos - builds that need to be made permanently available have to be put on that node.

IIRC this is a long-standing issue with the CI infrastructure. The upgrade suite yaml references (and needs to reference) builds by tag.
Older tags like v12.2.2 work fine whereas newer ones (v12.2.5 in this case) do not.

My unverified hypothesis is this: "The reason v12.2.2 works is because the build of the corresponding SHA1 cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba was placed on the 'special' chacra node that doesn't autodelete."

Okay, this is weird:
v12.2.2 is cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba
v12.2.5 is cad919881333ac92274171586c827e01f554a70a

https://chacra.ceph.com/binaries/ceph/luminous/cad919881333ac92274171586c827e01f554a70a/ yields a result
https://chacra.ceph.com/binaries/ceph/luminous/cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba/ also does

https://shaman.ceph.com/api/search/?sha1=cad919881333ac92274171586c827e01f554a70a does not yield a result
https://shaman.ceph.com/api/search/?sha1=cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba does, and it's a build on chacra.ceph.com from 2017-11-30.

The question remains - what is the easiest and correct way to address this: build system or teuthology?

From testing standpoint it's nightmare when released versions become unavailable and all tests fail :(

Nathan Cutler wrote:

Yuri Weinstein wrote:

What is we had a lookup table with hardcoded values for download.ceph.com, e.g.:

v12.2.2 --- sha1xxx
v12.2.5 --- sha1yyy
v10.0.10 --- sha1zzz
....

they won't change, will they ?

No, they won't change, but this would merely replace one manual process (putting the build of a particular SHA1 onto a special chacra node that doesn't delete the repo after a short period of time) with another one (adding entries to the lookup table).

It seems like this is a trivial table to build (in both directions) just by listing the tags.

with yum you can install specific version even if the repo has 12.2.4 and 12.2.5, although i am not sure how dependency pkgs will get picked up(but shouldn't be a big deal)

we have the same issue with cdn for point releases and I had request for ceph-ansible to handle installing specific version.

v12.2.0 32ce2a3ae5239ee33d6150705cdb24d43bab910c
v12.2.1 3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e
v12.2.2 cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba
v12.2.3 2dab17a455c09584f2a85e6b10888337d1ec8949
v12.2.4 52085d5249a80c5f5121a76d6288429f35e4e77b
v12.2.5 cad919881333ac92274171586c827e01f554a70a
v12.2.6 488df8a1076c4f5fc5b8d18a90463262c438740f

These are the SHA1s that, for Luminous so far, we need to have builds for on the "special Chacra node that doesn't delete its repos".

As part of our release workflow, we could open a ticket for the CI team to make sure each successive released SHA1 is added to that Chacra node.

Yuri, would this work for you?

I think it will work, Nathan.

We need to add `mimic` sha1 as well

David, what do you think?

These builds are on the special Chacra node that doesn't delete builds.

https://chacra.ceph.com/r/ceph/luminous/
https://chacra.ceph.com/r/ceph/mimic/

The problem is there is currently no functionality for teuthology to use the repos on chacra.ceph.com because there is no Shaman instance for chacra.ceph.com.

I see a few solutions here:

1. Code teuthology to go from tag -> sha1 -> chacra.ceph.com URL and use those repos if shaman.ceph.com returns nothing
2. Set up a separate shaman instance for chacra.ceph.com (?)

David Galloway wrote:

I see a few solutions here:

1. Code teuthology to go from tag -> sha1 -> chacra.ceph.com URL and use those repos if shaman.ceph.com returns nothing

This could work if chacra supported sha1 as a query parameter at /search/.

But I still strongly feel that the correct solution would be shaman becoming aware of chacra.ceph.com.

Zack Cerza wrote:

David Galloway wrote:

I see a few solutions here:

1. Code teuthology to go from tag -> sha1 -> chacra.ceph.com URL and use those repos if shaman.ceph.com returns nothing

This could work if chacra supported sha1 as a query parameter at /search/.

But I still strongly feel that the correct solution would be shaman becoming aware of chacra.ceph.com.

This would be feasible if we could tell shaman not to use chacra.ceph.com for dev builds. That would require coding in shaman and chacra however.

Can we discuss this in standup this week (infra standup monday?) and come to a consensus on the solution?

This gap is one of the contributing factors to the 12.2.6 debacle.

One alternative that might work is to enable the HTTP callback/reporting from the production chacra instance (chacra.ceph.com) to shaman. And somehow make shaman understand that there are these special repos that don't go away.

I think that Shaman doesn't actually purge anything unless it is being told to, so it is possible that just enabling reporting from chacra.ceph.com should suffice. Some spelunking into
both these systems (and their configs) would need to happen to really determine what is possible here.

Failing to have that working, maybe some sort of ad-hoc solution for releases only into shaman could be tried.

@David Galloway

We need to have ATM:
`luminous v12.2.2`
`luminous v12.2.5`
`luminous v12.2.7`

`mimic v13.2.0`
`mimic v13.2.1`
and future points