Project

General

Profile

Backport #19080

"TestLibRBD.UpdateFeatures" tests failed in upgrade:client-upgrade-jewel-distro-basic-vps

Added by Yuri Weinstein over 2 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Target version:
-
Release:
jewel

Description

Fixed by David Galloway in #18089 - see https://github.com/ceph/ceph/pull/14459 for more information.

History

#1 Updated by Yuri Weinstein over 2 years ago

  • Related to Bug #16898: "TestLibRBD.UpdateFeatures" tests failed in upgrade:client-upgrade-infernalis-client-x run added

#2 Updated by Mykola Golub over 2 years ago

This is the problem with the test. And the fix for this test was merged to infernalis branch long time ago:

commit baf17c90980f3c61f75775f561ced5b2a1d2141c
Merge: cfdea3e 0ee705f
Author: Mykola Golub <mgolub@mirantis.com>
Date:   Wed Nov 2 08:42:35 2016 +0200

    Merge pull request #10713 from dillaman/wip-16898-infernalis

    test: librbd features test case should only disable active features

    Reviewed-by: Mykola Golub <mgolub@mirantis.com>

(this is currently the last commit to infernalis branch).

The problem is that the suite installs infernalis packages using sha 752b6a3, i.e. for 9.2.1, which does not include this fix.

2017-02-24T17:58:12.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
2017-02-24T17:58:13.127 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.128 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty

#3 Updated by Nathan Cutler over 2 years ago

  • Assignee set to Nathan Cutler

#4 Updated by Nathan Cutler over 2 years ago

Indeed. Teuthology correctly recognizes the infernalis SHA1 that is needed, but when it queries Shaman it only asks for "infernalis".

2017-02-24T17:58:12.770 INFO:teuthology.packaging:branch: infernalis
2017-02-24T17:58:12.770 INFO:teuthology.packaging:sha1: d9eaab456ff45ae88e83bd633f0c4efb5902bf07
2017-02-24T17:58:12.771 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
...

I guess the problem is here (even though teuthology knows the specific SHA1 that is needed, it specifies "ref: None". This is the same as telling Shaman "Give me any infernalis build; I don't care which one"):

2017-02-24T17:58:12.854 INFO:teuthology.packaging:ref: None
2017-02-24T17:58:12.855 INFO:teuthology.packaging:tag: None
2017-02-24T17:58:12.856 INFO:teuthology.packaging:branch: infernalis

Shaman gives it the last released infernalis (9.2.1) because it satisfies the query:

2017-02-24T17:58:12.857 INFO:teuthology.packaging:sha1: d9eaab456ff45ae88e83bd633f0c4efb5902bf07
2017-02-24T17:58:12.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=infernalis
2017-02-24T17:58:13.127 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.128 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty
2017-02-24T17:58:13.176 INFO:teuthology.task.install.deb:Pulling from https://chacra.ceph.com/r/ceph/infernalis/752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd/ubuntu/trusty/flavors/default/
2017-02-24T17:58:13.180 INFO:teuthology.task.install.deb:Package version is 9.2.1-1trusty

#5 Updated by Nathan Cutler over 2 years ago

  • Project changed from rbd to teuthology

#6 Updated by Nathan Cutler over 2 years ago

One option is to modify the test so it specifies the exact SHA1 (but then it won't get any newer commits that might be added to infernalis later). Another option is to fix teuthology so it prefers "Ref: d9eaab456ff45ae88e83bd633f0c4efb5902bf07" over "Branch: infernalis" when deciding how to query Shaman. The latter is more robust, obviously.

However, no matter which option is chosen/implemented, the job will still fail because Shaman almost certainly does not have a build of d9eaab456ff45ae88e83bd633f0c4efb5902bf07 (because that SHA1 is not an official release). We could ask Shaman to build that SHA1, and it will comply, but then about 10-14 days later it will delete the build and there is no way to prevent that from happening - see #18089.

#7 Updated by Yuri Weinstein over 2 years ago

@Nathan - how would you like to finish it for 10.2.6 point release ?

#8 Updated by Nathan Cutler over 2 years ago

@Nathan - how would you like to finish it for 10.2.6 point release ?

There are only two options: either ignore the upgrade test failures, or fix them. This failure is just one of a number of varying instances of #18089 which needs to be fixed by the CI team. (It sounds like they could put the legacy builds that are used in the upgrade test on a Chacra host that is not configured to delete builds after a certain number of days.)

Once the Chacra issue is fixed, it will be trivial to edit the test YAML so it specifies a SHA1 instead of just "branch: infernalis".

#9 Updated by Nathan Cutler over 2 years ago

Chacra issue has been fixed, asked David in #18089 to add the builds

#10 Updated by Nathan Cutler over 2 years ago

  • Tracker changed from Bug to Backport
  • Project changed from teuthology to Ceph
  • Status changed from New to In Progress

#11 Updated by Nathan Cutler over 2 years ago

  • Description updated (diff)

#12 Updated by Nathan Cutler over 2 years ago

original description

This is for jewel v10.2.6 point release

Run: http://pulpito.front.sepia.ceph.com/yuriw-2017-02-24_17:46:07-upgrade:client-upgrade-jewel-distro-basic-vps/
Jobs: '856515', '856504'
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2017-02-24_17:46:07-upgrade:client-upgrade-jewel-distro-basic-vps/856515/teuthology.log

2017-02-24T18:03:14.149 INFO:tasks.workunit.client.0.vpm109.stdout:using new format!
2017-02-24T18:03:14.215 INFO:tasks.workunit.client.0.vpm109.stdout:test/librbd/test_librbd.cc:3029: Failure
2017-02-24T18:03:14.219 INFO:tasks.workunit.client.0.vpm109.stdout:Value of: image.update_features((1<<2) | (1<<3) | (1<<4), false)
2017-02-24T18:03:14.220 INFO:tasks.workunit.client.0.vpm109.stdout: Actual: -22
2017-02-24T18:03:14.220 INFO:tasks.workunit.client.0.vpm109.stdout:Expected: 0
2017-02-24T18:03:14.233 INFO:tasks.workunit.client.0.vpm109.stdout:[ FAILED ] TestLibRBD.UpdateFeatures (86 ms)

#13 Updated by Nathan Cutler over 2 years ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved

no target version because the fix did not require any code changes

#14 Updated by Nathan Cutler about 2 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF