Ceph

This is the problem with the test. And the fix for this test was merged to infernalis branch long time ago:

commit baf17c90980f3c61f75775f561ced5b2a1d2141c
Merge: cfdea3e 0ee705f
Author: Mykola Golub <mgolub@mirantis.com>
Date:   Wed Nov 2 08:42:35 2016 +0200

    Merge pull request #10713 from dillaman/wip-16898-infernalis

    test: librbd features test case should only disable active features

    Reviewed-by: Mykola Golub <mgolub@mirantis.com>

(this is currently the last commit to infernalis branch).

The problem is that the suite installs infernalis packages using sha 752b6a3, i.e. for 9.2.1, which does not include this fix.

2017-02-24T17:58:12.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
2017-02-24T17:58:13.127 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.128 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty

Indeed. Teuthology correctly recognizes the infernalis SHA1 that is needed, but when it queries Shaman it only asks for "infernalis".

2017-02-24T17:58:12.770 INFO:teuthology.packaging:branch: infernalis
2017-02-24T17:58:12.770 INFO:teuthology.packaging:sha1: d9eaab456ff45ae88e83bd633f0c4efb5902bf07
2017-02-24T17:58:12.771 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
...

I guess the problem is here (even though teuthology knows the specific SHA1 that is needed, it specifies "ref: None". This is the same as telling Shaman "Give me any infernalis build; I don't care which one"):

2017-02-24T17:58:12.854 INFO:teuthology.packaging:ref: None
2017-02-24T17:58:12.855 INFO:teuthology.packaging:tag: None
2017-02-24T17:58:12.856 INFO:teuthology.packaging:branch: infernalis

Shaman gives it the last released infernalis (9.2.1) because it satisfies the query:

2017-02-24T17:58:12.857 INFO:teuthology.packaging:sha1: d9eaab456ff45ae88e83bd633f0c4efb5902bf07
2017-02-24T17:58:12.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
2017-02-24T17:58:13.127 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.128 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty
2017-02-24T17:58:13.176 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.180 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty

One option is to modify the test so it specifies the exact SHA1 (but then it won't get any newer commits that might be added to infernalis later). Another option is to fix teuthology so it prefers "Ref: d9eaab456ff45ae88e83bd633f0c4efb5902bf07" over "Branch: infernalis" when deciding how to query Shaman. The latter is more robust, obviously.

However, no matter which option is chosen/implemented, the job will still fail because Shaman almost certainly does not have a build of d9eaab456ff45ae88e83bd633f0c4efb5902bf07 (because that SHA1 is not an official release). We could ask Shaman to build that SHA1, and it will comply, but then about 10-14 days later it will delete the build and there is no way to prevent that from happening - see #18089.

@Nathan Weinberg - how would you like to finish it for 10.2.6 point release ?

@Nathan Weinberg - how would you like to finish it for 10.2.6 point release ?

There are only two options: either ignore the upgrade test failures, or fix them. This failure is just one of a number of varying instances of #18089 which needs to be fixed by the CI team. (It sounds like they could put the legacy builds that are used in the upgrade test on a Chacra host that is not configured to delete builds after a certain number of days.)

Once the Chacra issue is fixed, it will be trivial to edit the test YAML so it specifies a SHA1 instead of just "branch: infernalis".

Chacra issue has been fixed, asked David in #18089 to add the builds

original description¶

This is for jewel v10.2.6 point release

Run: http://pulpito.front.sepia.ceph.com/yuriw-2017-02-24_17:46:07-upgrade:client-upgrade-jewel-distro-basic-vps/
Jobs: '856515', '856504'
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2017-02-24_17:46:07-upgrade:client-upgrade-jewel-distro-basic-vps/856515/teuthology.log

2017-02-24T18:03:14.149 INFO:tasks.workunit.client.0.vpm109.stdout:using new format!
2017-02-24T18:03:14.215 INFO:tasks.workunit.client.0.vpm109.stdout:test/librbd/test_librbd.cc:3029: Failure
2017-02-24T18:03:14.219 INFO:tasks.workunit.client.0.vpm109.stdout:Value of: image.update_features((1<<2) | (1<<3) | (1<<4), false)
2017-02-24T18:03:14.220 INFO:tasks.workunit.client.0.vpm109.stdout: Actual: -22
2017-02-24T18:03:14.220 INFO:tasks.workunit.client.0.vpm109.stdout:Expected: 0
2017-02-24T18:03:14.233 INFO:tasks.workunit.client.0.vpm109.stdout:[ FAILED ] TestLibRBD.UpdateFeatures (86 ms)