Bug #9769: upgrade/firefly: latest_dumpling_release.yaml always fails - Ceph - Ceph

Actions

Copy link

Bug #9769

closed

upgrade/firefly: latest_dumpling_release.yaml always fails

Added by Sage Weil over 9 years ago. Updated over 9 years ago.

Status:

Resolved

Priority:

Immediate

Assignee:

Yuri Weinstein

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

ubuntu@teuthology:/a/sage-2014-10-13_20:46:44-upgrade:firefly-firefly-distro-basic-multi$ teuthology-ls . | grep latest
546614 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 2090s
546615 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 2000s
546616 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 2257s
546617 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 2478s
546618 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 357s
546619 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 468s
546620 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 489s
546621 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 595s
546622 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 1776s
546623 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 2076s
546624 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 2343s
546625 FAIL scheduled_sage@flab upgrade:firefly/older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_14.04.yaml} 2141s

Actions

Copy link

Updated by Yuri Weinstein over 9 years ago

Status changed from New to In Progress

Running with verbose is on for job - http://qa-proxy.ceph.com/teuthology/sage-2014-10-13_20:46:44-upgrade:firefly-firefly-distro-basic-multi/546614/

Actions

Copy link

Updated by Yuri Weinstein over 9 years ago

From rerun with verbose:

2014-10-14T09:02:10.187 INFO:teuthology.orchestra.run.vpm195:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2014-10-14T09:02:10.371 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN crush map has legacy tunables
2014-10-14T09:02:11.371 ERROR:teuthology.parallel:Exception in parallel execution

Will add to 1-install/latest_dumpling_release.yaml:

overrides:
  ceph:
     conf:
       mon:
          mon warn on legacy crush tunables: false

Actions

Copy link

Updated by Yuri Weinstein over 9 years ago

Tests still failing but at different point.
See http://qa-proxy.ceph.com/teuthology/teuthology-2014-10-14_14:22:47-upgrade:firefly:older-firefly-distro-basic-vps/548193/teuthology.log

2014-10-14T14:39:16.886 INFO:tasks.workunit.client.0.vpm012.stderr:+ rbd ls -l
2014-10-14T14:39:17.001 INFO:tasks.workunit:Stopping rbd/import_export.sh on client.0...
2014-10-14T14:39:17.001 INFO:teuthology.orchestra.run.vpm012:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2014-10-14T14:39:17.371 ERROR:teuthology.parallel:Exception in parallel execution

I suspect this is because we should not be running import_export.sh while upgrading clients.

Put clients on separate nodes - commit/ea465ea14a53046c975b93992cc1689d711f5bac
Branch ceph-qa-tests - wip-9769

Scheduled new run http://pulpito.front.sepia.ceph.com/teuthology-2014-10-14_16:49:05-upgrade:firefly:older-firefly-distro-basic-vps/

Actions

Copy link

Updated by Yuri Weinstein over 9 years ago

Status changed from In Progress to 7

It fixed, testing now, here is the run passed:

teuthology@teuthology:~$ teuthology-ls /a/teuthology-2014-10-15_16:43:53-upgrade:firefly:older-firefly-distro-basic-vps | grep latest | grep pass
550193 pass yuriw upgrade:firefly:older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 3888s
550195 pass yuriw upgrade:firefly:older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 3844s
550197 pass yuriw upgrade:firefly:older/{0-cluster/start.yaml 1-install/latest_dumpling_release.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml} 4039s

Moved mds upgrade step after ods step - https://github.com/ceph/ceph-qa-suite/pull/194

Actions

Copy link