Project

General

Profile

Tasks #17151

hammer v0.94.10

Added by Nathan Cutler over 1 year ago. Updated 10 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
Start date:
08/26/2016
Due date:
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Release:
Needs Doc:
No

Description

Workflow

  • Preparing the release
  • Cutting the release
    • Nathan asks Abhishek (Release Manager) if a point release should be published - YES
    • Nathan gets approval from all leads
      • Yehuda, rgw - YES
      • John, CephFS - YES
      • Jason, RBD - YES
      • Josh, rados - YES
    • Nathan prepares draft release notes -DONE https://github.com/ceph/ceph/pull/13152
      • ceph-objectstore-tool and ceph-monstore-tool now enable user to rebuild the monitor database from OSDs. This feature is especially useful when all monitors fail to boot due to leveldb corruption.
    • Abhishek writes and commits the release notes
    • Nathan informs Yuri that the branch is ready for testing - DONE
    • Yuri runs additional integration tests - DONE
    • If Yuri discovers new bugs that need to be backported urgently (i.e. their priority is set to Urgent), the release goes back to being prepared, it was not ready after all
    • Yuri informs Abhishek that the branch is ready for release - DONE
    • Abhishek creates the packages and sets the release tag - IN PROGRESS

Release information

History

#1 Updated by Nathan Cutler over 1 year ago

  • Subject changed from hammer v0.94.9 to hammer v0.94.10
  • Target version changed from v0.94.9 - UNDELETABLE, DO NOT USE to 522

0.94.9 is out already. Repurposing this issue for 0.94.10.

#2 Updated by Nathan Cutler about 1 year ago

  • Description updated (diff)
  • Needs Doc set to No

#3 Updated by Nathan Cutler about 1 year ago

  • Description updated (diff)

#4 Updated by Nathan Cutler about 1 year ago

  • Description updated (diff)

#5 Updated by Nathan Cutler about 1 year ago

  • Status changed from New to In Progress

#6 Updated by Nathan Cutler about 1 year ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#7 Updated by Nathan Cutler about 1 year ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 20)/20 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:40:41-rados-hammer-backports---basic-smithi/

  • new bug http://tracker.ceph.com/issues/17955 rados/test_pool_quota.sh fails
    • 558016 rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml}
    • 558140 rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}
    • David Zafman found that the test failures are due to the new "pool full" detection/handling in the Objecter: "it's a feature, not a bug" - just need to fix the test
  • known bug http://tracker.ceph.com/issues/15139 Command failed on smithi014 with status 1: "sudo yum -y install '' ceph-radosgw"
  • known bug http://tracker.ceph.com/issues/15345 ./common/RWLock.h: In function 'void RWLock::get_write(bool)' ./common/RWLock.h: 97: FAILED assert(r == 0)
    • 558164 rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/fastclose.yaml supported/ubuntu_14.04.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}
    • 558237 rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/readwrite.yaml}
    • 558253 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/pool-snaps-few-objects.yaml}
    • addressed by staging backport https://github.com/ceph/ceph/pull/12071

Pushed hammer-backports-12071 branch to ceph/ceph and re-running to see if PR#12071 fixes http://tracker.ceph.com/issues/15345:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071 --machine-type smithi --filter 'rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/fastclose.yaml supported/ubuntu_14.04.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml},rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/readwrite.yaml},rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/pool-snaps-few-objects.yaml}'

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_19:36:26-rados-hammer-backports-12071---basic-smithi/

Re-running the test_pool_quota.sh failures on hammer-backports-12071-17955 (which includes cherry-pick of 16ead95daa3d1309e8e76e57416b4201e71d0449):

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071-17955 --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}'

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_10:47:45-rados-hammer-backports-12071-17955---basic-smithi/

#8 Updated by Nathan Cutler about 1 year ago

RADOS BASELINE

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 20)/20 --suite-branch hammer --email ncutler@suse.cz --ceph hammer --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:47:09-rados-hammer---basic-smithi/

#9 Updated by Nathan Cutler about 1 year ago

powercycle

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:57:30-powercycle-hammer-backports-testing-basic-smithi/
  • ceph packages failed to install
  • Loic had the same problem with jewel - the issue is that the suite has changed in ceph-qa-suite master, so we need to run with --suite-branch hammer

Second try:

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k testing -m smithi -s powercycle -p 1000 --email ncutler@suse.cz --suite-branch hammer

one passed, one dead http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_15:27:08-powercycle-hammer-backports-testing-basic-smithi/

Repeat the dead test:

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k testing -m smithi -s powercycle -p 1000 --email ncutler@suse.cz --suite-branch hammer --filter 'powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}'

dead http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_21:53:44-powercycle-hammer-backports-testing-basic-smithi/

#10 Updated by Nathan Cutler about 1 year ago

fs

teuthology-suite --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/

Re-running all five dead and failed jobs:

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/

#11 Updated by Nathan Cutler about 1 year ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_10:00:20-rgw-hammer-backports---basic-smithi/

Re-running the failed test 5 times:

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:12:20-rgw-hammer-backports---basic-smithi/

#12 Updated by Nathan Cutler about 1 year ago

rbd

teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_10:01:33-rbd-hammer-backports---basic-smithi/

Re-running both jobs:

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_22:58:18-rbd-hammer-backports---basic-smithi/

#13 Updated by Nathan Cutler about 1 year ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#14 Updated by Nathan Cutler about 1 year ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_11:33:58-rados-hammer-backports---basic-smithi/

(pool quota tests failed because the run somehow picked up the "old" hammer-backports branch, i.e. 2dd92e79592d27661e0b13d1da5522da995a187a which did not include the fix)

Re-running the pool quota tests 10 times:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071-17955 --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}' -N 10

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:07:22-rados-hammer-backports-12071-17955---basic-smithi/

And a slightly modified run of the same two tests:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}' -N 5

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:13:55-rados-hammer-backports---basic-smithi/

#15 Updated by Nathan Cutler about 1 year ago

rados - PR#10255

A 5x run to assert that a failure from https://github.com/ceph/ceph/pull/10255#issuecomment-241138225 is not reproducible:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/cache.yaml}' -N 5

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:24:11-rados-hammer-backports---basic-smithi/

#16 Updated by Nathan Cutler about 1 year ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#17 Updated by Nathan Cutler about 1 year ago

rados

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_11:52:41-rados-hammer-backports---basic-smithi/

  • old dumpling, firefly packages no longer available

Although the run is technically a "fail", there were only two failed tests, both caused by the inability to install very old (dumpling-, firefly-era) packages.

#18 Updated by Nathan Cutler about 1 year ago

upgrade

./virtualenv/bin/teuthology-suite -l 2 -k distro --verbose --suite upgrade/hammer --suite-branch hammer --ceph hammer-backports --machine-type vps --priority 1000 --dry-run machine_types/vps.yaml

pass http://pulpito.ceph.com/smithfarm-2016-11-21_11:59:49-upgrade:hammer-hammer-backports-distro-basic-vps/

Plus this one job:

pass http://pulpito.ceph.com/smithfarm-2016-11-21_11:57:05-upgrade:hammer-hammer-backports-distro-basic-vps/566665/

#19 Updated by Nathan Cutler about 1 year ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k distro -m smithi -s powercycle -p 90 --email ncutler@suse.cz --suite-branch hammer

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_23:01:04-powercycle-hammer-backports-distro-basic-smithi/

#20 Updated by Nathan Cutler about 1 year ago

  • Description updated (diff)

#21 Updated by Nathan Cutler about 1 year ago

  • Description updated (diff)

#22 Updated by Nathan Cutler about 1 year ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#23 Updated by Nathan Cutler about 1 year ago

rados

teuthology-suite --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:37:15-rados-hammer-backports---basic-smithi/

So, despite the one failure, the run should be considered a pass

#24 Updated by Nathan Cutler about 1 year ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:41:08-rgw-hammer-backports---basic-smithi/

#25 Updated by Nathan Cutler about 1 year ago

fs

teuthology-suite --priority 101 --suite fs --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}'

So we can assert that http://tracker.ceph.com/issues/14716 is not triggered.

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:42:11-fs-hammer-backports---basic-smithi/

Unfortunately, the failure is a case of http://tracker.ceph.com/issues/14716

#26 Updated by Nathan Cutler about 1 year ago

fs

Pushed hammer-backports-14716 with revert of http://github.com/ceph/ceph/commit/4a36933 which was recently merged to hammer and might be causing this test to fail:

teuthology-suite --priority 101 --suite fs --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports-14716 --machine-type smithi --filter 'recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}'

dead http://pulpito.ceph.com:80/smithfarm-2016-11-28_10:51:30-fs-hammer-backports-14716---basic-smithi/

#27 Updated by Nathan Cutler about 1 year ago

  • Target version changed from 522 to v0.94.10

#28 Updated by Nathan Cutler 12 months ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#29 Updated by Nathan Cutler 11 months ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_11:16:57-rados-wip-hammer-backports-distro-basic-smithi/

Re-running with Xenial excluded:

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_12:24:43-rados-wip-hammer-backports-distro-basic-smithi/

Re-pushed new integration branch with https://github.com/ceph/ceph/pull/12743 and running the full 112 jobs again:

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_15:48:30-rados-wip-hammer-backports-distro-basic-smithi/ (four fail, three dead)

#30 Updated by Nathan Cutler 11 months ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_15:51:41-rgw-wip-hammer-backports-distro-basic-smithi/

Re-running failed tests with --suite-repo https://github.com/ceph/ceph.git --suite-branch hammer (to avoid issue#18384):

./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --suite-repo https://github.com/ceph/ceph.git --suite-branch hammer --filter="$filter" qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-02_19:50:30-rgw-wip-hammer-backports-distro-basic-smithi/

#31 Updated by Nathan Cutler 11 months ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:38:11-rbd-wip-hammer-backports-distro-basic-smithi/ (two fail, one dead)

Re-running dead and failed jobs with fix for issue#18388

teuthology-suite --priority 101 --suite rbd --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="$filter" qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:52:46-rbd-wip-hammer-backports-distro-basic-smithi/

#32 Updated by Nathan Cutler 11 months ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:39:27-fs-wip-hammer-backports-distro-basic-smithi/ (one dead job)

#33 Updated by Nathan Cutler 11 months ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:40:28-powercycle-wip-hammer-backports-distro-basic-smithi/

#34 Updated by Nathan Cutler 11 months ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#35 Updated by Nathan Cutler 11 months ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:23:21-rgw-wip-hammer-backports-distro-basic-smithi/

#36 Updated by Nathan Cutler 11 months ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

running http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:24:28-powercycle-wip-hammer-backports-distro-basic-smithi/

#37 Updated by Nathan Cutler 11 months ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#38 Updated by Nathan Cutler 11 months ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-03_20:59:45-rados-wip-hammer-backports-distro-basic-smithi/

Re-running two failed (centos-only) jobs without "qa/distros/all/ubuntu_14.04.yaml":

one pass, one stuck in "queued" state http://pulpito.ceph.com:80/smithfarm-2017-01-04_19:05:09-rados-wip-hammer-backports-distro-basic-smithi/

Re-running one failed job with "qa/distros/all/ubuntu_14.04.yaml":

pass http://pulpito.ceph.com:80/smithfarm-2017-01-04_19:06:53-rados-wip-hammer-backports-distro-basic-smithi/

Re-running one job that seemed to get stuck in "Queued" state:

pass http://pulpito.ceph.com:80/smithfarm-2017-01-04_20:52:15-rados-wip-hammer-backports-distro-basic-smithi/

Re-running last job that got stuck in "Queued" state:

pass http://pulpito.ceph.com:80/smithfarm-2017-01-05_10:54:32-rados-wip-hammer-backports-distro-basic-smithi/

#39 Updated by Nathan Cutler 11 months ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:01:35-rgw-wip-hammer-backports-distro-basic-smithi/

#40 Updated by Nathan Cutler 11 months ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:02:28-rbd-wip-hammer-backports-distro-basic-smithi/

#41 Updated by Nathan Cutler 11 months ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:09:18-powercycle-wip-hammer-backports-distro-basic-smithi/

#42 Updated by Nathan Cutler 11 months ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:09:50-fs-wip-hammer-backports-distro-basic-smithi/

Though technically "fail", this run is actually "pass" because the only failure can safely be ignored.

#44 Updated by Nathan Cutler 11 months ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-05_20:47:44-rgw-wip-hammer-backports-distro-basic-smithi/ (one failed job)

Re-running the failed job:

teuthology-suite --priority 101 --suite rgw --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/btrfs.yaml overrides.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}" --dry-run qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-06_16:38:29-rgw-wip-hammer-backports-distro-basic-smithi/

#46 Updated by Nathan Cutler 11 months ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-13_18:40:44-rados-wip-hammer-backports-distro-basic-smithi/

  • failed jobs - not expected to succeed (?) but fix might be feasible, see http://tracker.ceph.com/issues/18069#note-7
    • rados/singleton-nomsgr/{all/11429.yaml}
    • rados/singleton-nomsgr/{all/13234.yaml}
  • plus three tests that need to be re-run without "qa/distros/all/ubuntu_14.04.yaml" because teuthology thinks they should be run on "centos 14.04"

Re-running the latter three failed tests without qa/distros/all/ubuntu_14.04.yaml

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="$filter"

pass http://pulpito.ceph.com:80/smithfarm-2017-01-20_20:54:24-rados-wip-hammer-backports-distro-basic-smithi/

#47 Updated by Nathan Cutler 11 months ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-13_18:43:12-powercycle-wip-hammer-backports-distro-basic-smithi/

#48 Updated by Nathan Cutler 11 months ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#49 Updated by Nathan Cutler 11 months ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-20_23:47:14-rbd-wip-hammer-backports-distro-basic-smithi/

#50 Updated by Nathan Cutler 11 months ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-20_23:49:04-fs-wip-hammer-backports-distro-basic-smithi/

#51 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#52 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#53 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#54 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#55 Updated by Nathan Cutler 11 months ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

#56 Updated by Nathan Cutler 11 months ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-21_15:59:00-fs-wip-hammer-backports-distro-basic-smithi/

  • The only failure is fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml} which was not expected to pass

Ruled a pass

#57 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#58 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#59 Updated by Nathan Cutler 11 months ago

  • Description updated (diff)

#60 Updated by Yuri Weinstein 11 months ago

  • Description updated (diff)

#61 Updated by Yuri Weinstein 11 months ago

QE VALIDATION (STARTED 1/23/17)

re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v0.94.10_QE_validation_notes
COMMAND LINE => CEPH_BRANCH=83af8cdaaa6d94404e6146b68e532a784e3cc99c; MACHINE_NAME=vps; teuthology-suite -v -S $CEPH_BRANCH -m $MACHINE_NAME -k distro -s rados -e $CEPH_QA_EMAIL --suite-branch hammer

(Note: PASSED / FAILED - indicates "TEST IS IN PROGRESS")

All suites won't test on ubuntu 16.04 http://tracker.ceph.com/issues/18382

The following tests are not expected to pass:

  • fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml} -> http://tracker.ceph.com/issues/14716
  • rados/singleton-nomsgr/{all/11429.yaml}
  • rados/singleton-nomsgr/{all/13234.yaml}
Suite Runs/Reruns Notes/Issues
rados ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_16:20:34-rados-master-distro-basic-vps/ FAILED -subset $(expr $RANDOM % 20)/20
http://pulpito.ceph.com/teuthology-2017-01-26_17:48:56-rados-hammer---basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-28_16:54:09-rados-hammer---basic-smithi/ approved by Josh
rados centos http://pulpito.ceph.com/teuthology-2017-01-25_16:32:25-rados-master-distro-basic-vps/ approved by Josh
http://pulpito.ceph.com/teuthology-2017-01-26_17:52:58-rados-hammer---basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-28_16:55:30-rados-hammer---basic-smithi/ approved by Josh
rbd ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_22:00:24-rbd-master-distro-basic-smithi/ ubuntu PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_17:28:55-rbd-hammer---basic-mira/
rbd centos http://pulpito.ceph.com/teuthology-2017-01-25_22:04:27-rbd-master-distro-basic-smithi/ centos PASSED
rgw ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_22:18:36-rgw-master-distro-basic-smithi/ ubuntu PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_17:22:10-rgw-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-28_16:26:40-rgw-hammer---basic-smithi/
rgw centos http://pulpito.ceph.com/teuthology-2017-01-25_22:21:30-rgw-master-distro-basic-smithi/ centos FAILED #18735
http://pulpito.ceph.com/teuthology-2017-01-26_17:19:58-rgw-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-28_16:28:47-rgw-hammer---basic-smithi/ Approval/decision by Yehuda/Sage needed
http://pulpito.ceph.com/teuthology-2017-01-28_22:35:40-rgw-hammer---basic-vps/ trying centos 7.2
http://pulpito.ceph.com/teuthology-2017-02-14_16:21:46-rgw-hammer-distro-basic-smithi/
retesting with fix http://tracker.ceph.com/issues/18744
fs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_18:02:24-fs-master-distro-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by John
"The MDL_WriteError one is the bug from when ENOSPC stuff was
backported wrongly, the FileJournal/FileStore assertions are from btrfs."
http://pulpito.ceph.com/teuthology-2017-01-28_16:31:30-fs-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-31_17:18:26-fs-hammer-distro-basic-smithi/ run off gitbuilders
fs centos http://pulpito.ceph.com/teuthology-2017-01-26_18:03:38-fs-master-distro-basic-smithi/ FAILED noise? approved by John
http://pulpito.ceph.com/teuthology-2017-01-28_16:32:30-fs-hammer---basic-smithi/
krbd ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_20:58:56-krbd-master-testing-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by Ilya
http://pulpito.ceph.com/teuthology-2017-01-28_16:38:13-krbd-hammer-testing-basic-smithi/
krbd centos http://pulpito.ceph.com/teuthology-2017-01-26_21:01:15-krbd-master-testing-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by Ilya
http://pulpito.ceph.com/teuthology-2017-01-28_16:38:58-krbd-hammer-testing-basic-smithi/
kcephfs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:03:50-kcephfs-master-testing-basic-smithi/
PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:40:15-kcephfs-hammer-testing-basic-smithi/
kcephfs centos http://pulpito.ceph.com/teuthology-2017-01-26_21:05:21-kcephfs-master-testing-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:41:05-kcephfs-hammer-testing-basic-smithi/
knfs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:08:14-knfs-master-testing-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:42:08-knfs-hammer-testing-basic-smithi/
knfs centos http://pulpito.ceph.com/teuthology-2017-01-26_21:08:46-knfs-master-testing-basic-smithi/ PASSED
hadoop ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:09:55-hadoop-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:43:53-hadoop-hammer---basic-smithi/
hadoop centos http://pulpito.ceph.com/teuthology-2017-01-26_21:10:40-hadoop-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:45:27-hadoop-hammer---basic-smithi/
samba ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:11:17-samba-master-distro-basic-smithi/ FAILED shaman build issues approved by Sage
http://pulpito.ceph.com/teuthology-2017-01-30_18:15:33-samba-hammer-distro-basic-smithi/ run off gitbuilders, packaging issues
samba centos http://pulpito.ceph.com/teuthology-2017-01-26_21:11:58-samba-master-distro-basic-smithi/ FAILED shaman build issues approved by Sage
rest ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:12:36-rest-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:47:24-rest-hammer---basic-smithi/
rest centos http://pulpito.ceph.com/teuthology-2017-01-26_21:14:18-rest-master-distro-basic-smithi/ PASSED
ceph-deploy ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:15:14-ceph-deploy-master-distro-basic-vps/ FAILED #18737, #18736
http://pulpito.ceph.com/teuthology-2017-01-30_17:38:19-ceph-deploy-hammer-distro-basic-vps/
ceph-deploy centos http://pulpito.ceph.com/teuthology-2017-01-26_21:15:47-ceph-deploy-master-distro-basic-vps/ FAILED same as 0.94.08 see #15895
upgrade/client-upgrade ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:29:02-upgrade:client-upgrade-master-distro-basic-vps/
upgrade/client-upgrade centos http://pulpito.ceph.com/teuthology-2017-01-26_21:30:09-upgrade:client-upgrade-master-distro-basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-30_20:48:08-upgrade:client-upgrade-hammer-distro-basic-vps/ run off gitbuilders a must! PASSED
upgrade/firefly-x (hammer) http://pulpito.ceph.com/teuthology-2017-01-30_20:52:09-upgrade:firefly-x-hammer-distro-basic-vps/ FAILED run off gitbuilders, approved by Sage
http://pulpito.ceph.com/teuthology-2017-01-31_16:19:21-upgrade:firefly-x-hammer---basic-smithi/
upgrade/hammer http://pulpito.ceph.com/teuthology-2017-01-26_21:34:09-upgrade:hammer-master-distro-basic-vps/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-30_20:57:57-upgrade:hammer-hammer-distro-basic-vps/ run off gitbuilders
http://pulpito.ceph.com/teuthology-2017-01-31_16:18:48-upgrade:hammer-hammer---basic-smithi/ 2 failed b/c of noise
upgrade/hammer-x (jewel) http://pulpito.ceph.com/teuthology-2017-02-07_18:15:23-upgrade:hammer-x-jewel-distro-basic-vps/ PASSED
http://pulpito.ceph.com/yuriw-2017-02-08_17:22:47-upgrade:hammer-x-jewel---basic-smithi/ run off gitbuilders
powercycle ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_16:42:38-powercycle-master-testing-basic-mira/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_16:47:48-powercycle-hammer---basic-mira/
powercycle centos http://pulpito.ceph.com/teuthology-2017-01-25_16:43:42-powercycle-master-testing-basic-mira/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_16:58:05-powercycle-hammer---basic-mira/
http://pulpito.ceph.com/teuthology-2017-01-28_16:50:25-powercycle-hammer-testing-basic-smithi/
(DON'T REMOVE BELOW)
Suite Runs/Reruns Notes/Issues
PASSED / FAILED

centos

#62 Updated by Nathan Cutler 11 months ago

rgw valgrind issues bisect

ubuntu ./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph hammer --machine-type smithi --ceph-repo http://github.com/ceph/ceph.git --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" qa/distros/all/ubuntu_14.04.yaml

centos 7.3 ./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph hammer --machine-type smithi --ceph-repo http://github.com/ceph/ceph.git --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" ~/os_centos.yaml

Tip of hammer ("bad" version for bisect) 83af8cdaaa6d94404e6146b68e532a784e3cc99c

Ubuntu 14.04

CentOS 7.3

0.94.9 (hopefully "good" version for bisect) wip-hammer-0-94-9

./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph wip-hammer-0-94-9 --machine-type smithi --ceph-repo http://github.com/ceph/ceph-ci.git --suite-branch hammer --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" ~/os_centos.yaml

Since the problem is present in the 0.94.9 release, we can be sure it's not due to a regression introduced in 0.94.10.

#63 Updated by Nathan Cutler 10 months ago

  • Description updated (diff)

#64 Updated by Nathan Cutler 10 months ago

(04:03:19 PM) sage: smithfarm: i think we can ignore hte notcmalloc hammer stuff and move on...
(04:03:59 PM) smithfarm: sage: ok, so afaict that means hammer 0.94.10 has passed qE
(04:04:33 PM) smithfarm: unless you think we should fix upgrade/firefly-x
(04:04:36 PM) sage: yeah i think so
(04:04:37 PM) sage: nope :)

#65 Updated by Abhishek Lekshmanan 10 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF