Project

General

Profile

Bug #19636

upgrade:client-upgrade/{hammer,jewel}-client-x/rbd failing in kraken 11.2.1 integration testing

Added by Nathan Cutler 5 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Target version:
-
Start date:
04/17/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

test descriptions:

  • upgrade:client-upgrade/hammer-client-x/rbd/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_notification_tests.yaml}
  • upgrade:client-upgrade/jewel-client-x/rbd/{0-cluster/start.yaml 1-install/jewel-client-x.yaml 2-workload/rbd_notification_tests.yaml}

test failure reproducible? YES

test URLs:

test cluster:

  roles:
  - - mon.a
    - mon.b
    - mon.c
    - osd.0
    - osd.1
    - osd.2
    - client.0
  - - client.1

what seems to happen:

  1. hammer is installed on both nodes
  2. the "client.1" node is upgraded to kraken (wip-kraken-backports)
  3. ceph task runs
  4. the "hammer" version of "rbd/notify_master.sh" is run on client.0 and the "hammer" version of "rbd/notify_slave.sh" is run on client.1 (afaict the hammer and kraken versions of these scripts are identical)
2017-04-16T18:44:20.785 INFO:tasks.workunit.client.0.vpm175.stderr:+
dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/notify_master.sh
2017-04-16T18:44:20.786 INFO:tasks.workunit.client.0.vpm175.stderr:+
relpath=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd
2017-04-16T18:44:20.787 INFO:tasks.workunit.client.0.vpm175.stderr:+
python /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../../src/test/librbd/test_notify.py
master

Three hours later, they are stopped (timeout):

2017-04-16T21:44:02.648 INFO:tasks.workunit:Stopping ['rbd/notify_master.sh'] on client.0...
2017-04-16T21:44:02.649 INFO:teuthology.orchestra.run.vpm089:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
2017-04-16T21:44:02.668 INFO:tasks.workunit:Stopping ['rbd/notify_slave.sh'] on client.1...
2017-04-16T21:44:02.669 INFO:teuthology.orchestra.run.vpm101:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1'

And the job fails with an exception:

CommandFailedError: Command failed (workunit test rbd/notify_master.sh) on vpm089 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=hammer TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=13 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/notify_master.sh'

Related issues

Copied to rbd - Backport #19659: kraken: upgrade:client-upgrade/{hammer,jewel}-client-x/rbd failing in kraken 11.2.1 integration testing Resolved

History

#1 Updated by Nathan Cutler 5 months ago

  • Description updated (diff)

#2 Updated by Nathan Cutler 5 months ago

  • Subject changed from upgrade:client-upgrade/hammer-client-x/rbd failing in kraken 11.2.1 integration testing to upgrade:client-upgrade/{hammer,jewel}-client-x/rbd failing in kraken 11.2.1 integration testing
  • Description updated (diff)

#3 Updated by Nathan Cutler 5 months ago

  • Description updated (diff)

#4 Updated by Jason Dillaman 5 months ago

Resize payload structure no longer ABI compatible after commit d1f2c557 (already in master and kraken branches).

#5 Updated by Jason Dillaman 5 months ago

  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman
  • Priority changed from Normal to High
  • Backport set to kraken

#6 Updated by Jason Dillaman 5 months ago

  • Status changed from In Progress to Need Review

#7 Updated by Mykola Golub 5 months ago

  • Status changed from Need Review to Pending Backport

#8 Updated by Nathan Cutler 5 months ago

  • Copied to Backport #19659: kraken: upgrade:client-upgrade/{hammer,jewel}-client-x/rbd failing in kraken 11.2.1 integration testing added

#9 Updated by Nathan Cutler 5 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF