Project

General

Profile

Actions
Copy link

Tasks #17151

closed

hammer v0.94.10

Added by Nathan Cutler over 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Nathan Cutler
Target version:
Ceph - v0.94.10
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

Workflow

  • Preparing the release
  • Cutting the release
    • Nathan asks Abhishek (Release Manager) if a point release should be published - YES
    • Nathan gets approval from all leads
      • Yehuda, rgw - YES
      • John, CephFS - YES
      • Jason, RBD - YES
      • Josh, rados - YES
    • Nathan prepares draft release notes -DONE https://github.com/ceph/ceph/pull/13152
      • ceph-objectstore-tool and ceph-monstore-tool now enable user to rebuild the monitor database from OSDs. This feature is especially useful when all monitors fail to boot due to leveldb corruption.
    • Abhishek writes and commits the release notes
    • Nathan informs Yuri that the branch is ready for testing - DONE
    • Yuri runs additional integration tests - DONE
    • If Yuri discovers new bugs that need to be backported urgently (i.e. their priority is set to Urgent), the release goes back to being prepared, it was not ready after all
    • Yuri informs Abhishek that the branch is ready for release - DONE
    • Abhishek creates the packages and sets the release tag - IN PROGRESS

Release information

Actions
Copy link
 #1

Updated by Nathan Cutler over 7 years ago

  • Subject changed from hammer v0.94.9 to hammer v0.94.10
  • Target version changed from v0.94.9 - UNDELETABLE, DO NOT USE to 522

0.94.9 is out already. Repurposing this issue for 0.94.10.

Actions
Copy link
 #2

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Nathan Cutler over 7 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #6

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #7

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 20)/20 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:40:41-rados-hammer-backports---basic-smithi/

  • new bug http://tracker.ceph.com/issues/17955 rados/test_pool_quota.sh fails
    • 558016 rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml}
    • 558140 rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}
    • David Zafman found that the test failures are due to the new "pool full" detection/handling in the Objecter: "it's a feature, not a bug" - just need to fix the test
  • known bug http://tracker.ceph.com/issues/15139 Command failed on smithi014 with status 1: "sudo yum -y install '' ceph-radosgw"
  • known bug http://tracker.ceph.com/issues/15345 ./common/RWLock.h: In function 'void RWLock::get_write(bool)' ./common/RWLock.h: 97: FAILED assert(r == 0)
    • 558164 rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/fastclose.yaml supported/ubuntu_14.04.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}
    • 558237 rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/readwrite.yaml}
    • 558253 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/pool-snaps-few-objects.yaml}
    • addressed by staging backport https://github.com/ceph/ceph/pull/12071

Pushed hammer-backports-12071 branch to ceph/ceph and re-running to see if PR#12071 fixes http://tracker.ceph.com/issues/15345:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071 --machine-type smithi --filter 'rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/fastclose.yaml supported/ubuntu_14.04.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml},rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/readwrite.yaml},rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/pool-snaps-few-objects.yaml}'

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_19:36:26-rados-hammer-backports-12071---basic-smithi/

Re-running the test_pool_quota.sh failures on hammer-backports-12071-17955 (which includes cherry-pick of 16ead95daa3d1309e8e76e57416b4201e71d0449):

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071-17955 --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}'

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_10:47:45-rados-hammer-backports-12071-17955---basic-smithi/

Actions
Copy link
 #8

Updated by Nathan Cutler over 7 years ago

RADOS BASELINE

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 20)/20 --suite-branch hammer --email ncutler@suse.cz --ceph hammer --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:47:09-rados-hammer---basic-smithi/

Actions
Copy link
 #9

Updated by Nathan Cutler over 7 years ago

powercycle

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:57:30-powercycle-hammer-backports-testing-basic-smithi/
  • ceph packages failed to install
  • Loic had the same problem with jewel - the issue is that the suite has changed in ceph-qa-suite master, so we need to run with --suite-branch hammer

Second try:

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k testing -m smithi -s powercycle -p 1000 --email ncutler@suse.cz --suite-branch hammer

one passed, one dead http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_15:27:08-powercycle-hammer-backports-testing-basic-smithi/

Repeat the dead test:

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k testing -m smithi -s powercycle -p 1000 --email ncutler@suse.cz --suite-branch hammer --filter 'powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}'

dead http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_21:53:44-powercycle-hammer-backports-testing-basic-smithi/

Actions
Copy link
 #10

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_09:59:00-fs-hammer-backports---basic-smithi/

Re-running all five dead and failed jobs:

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_22:39:17-fs-hammer-backports---basic-smithi/

Actions
Copy link
 #11

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_10:00:20-rgw-hammer-backports---basic-smithi/

Re-running the failed test 5 times:

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:12:20-rgw-hammer-backports---basic-smithi/

Actions
Copy link
 #12

Updated by Nathan Cutler over 7 years ago

rbd

teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-18_10:01:33-rbd-hammer-backports---basic-smithi/

Re-running both jobs:

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_22:58:18-rbd-hammer-backports---basic-smithi/
Actions
Copy link
 #13

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #14

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_11:33:58-rados-hammer-backports---basic-smithi/

(pool quota tests failed because the run somehow picked up the "old" hammer-backports branch, i.e. 2dd92e79592d27661e0b13d1da5522da995a187a which did not include the fix)

Re-running the pool quota tests 10 times:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports-12071-17955 --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}' -N 10

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:07:22-rados-hammer-backports-12071-17955---basic-smithi/

And a slightly modified run of the same two tests:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml},rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}' -N 5

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:13:55-rados-hammer-backports---basic-smithi/

Actions
Copy link
 #15

Updated by Nathan Cutler over 7 years ago

rados - PR#10255

A 5x run to assert that a failure from https://github.com/ceph/ceph/pull/10255#issuecomment-241138225 is not reproducible:

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/cache.yaml}' -N 5

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-20_23:24:11-rados-hammer-backports---basic-smithi/

Actions
Copy link
 #16

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #17

Updated by Nathan Cutler over 7 years ago

rados

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch wip-16225 --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_11:52:41-rados-hammer-backports---basic-smithi/

  • old dumpling, firefly packages no longer available

Although the run is technically a "fail", there were only two failed tests, both caused by the inability to install very old (dumpling-, firefly-era) packages.

Actions
Copy link
 #18

Updated by Nathan Cutler over 7 years ago

upgrade

./virtualenv/bin/teuthology-suite -l 2 -k distro --verbose --suite upgrade/hammer --suite-branch hammer --ceph hammer-backports --machine-type vps --priority 1000 --dry-run machine_types/vps.yaml

pass http://pulpito.ceph.com/smithfarm-2016-11-21_11:59:49-upgrade:hammer-hammer-backports-distro-basic-vps/

Plus this one job:

pass http://pulpito.ceph.com/smithfarm-2016-11-21_11:57:05-upgrade:hammer-hammer-backports-distro-basic-vps/566665/

Actions
Copy link
 #19

Updated by Nathan Cutler over 7 years ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports -k distro -m smithi -s powercycle -p 90 --email ncutler@suse.cz --suite-branch hammer

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_23:01:04-powercycle-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #20

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #21

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #22

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #23

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:37:15-rados-hammer-backports---basic-smithi/

So, despite the one failure, the run should be considered a pass

Actions
Copy link
 #24

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

pass http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:41:08-rgw-hammer-backports---basic-smithi/

Actions
Copy link
 #25

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi --filter 'recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}'

So we can assert that http://tracker.ceph.com/issues/14716 is not triggered.

fail http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-24_11:42:11-fs-hammer-backports---basic-smithi/

Unfortunately, the failure is a case of http://tracker.ceph.com/issues/14716

Actions
Copy link
 #26

Updated by Nathan Cutler over 7 years ago

fs

Pushed hammer-backports-14716 with revert of http://github.com/ceph/ceph/commit/4a36933 which was recently merged to hammer and might be causing this test to fail:

teuthology-suite --priority 101 --suite fs --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports-14716 --machine-type smithi --filter 'recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}'

dead http://pulpito.ceph.com:80/smithfarm-2016-11-28_10:51:30-fs-hammer-backports-14716---basic-smithi/

Actions
Copy link
 #27

Updated by Nathan Cutler over 7 years ago

  • Target version changed from 522 to v0.94.10
Actions
Copy link
 #28

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #29

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_11:16:57-rados-wip-hammer-backports-distro-basic-smithi/

Re-running with Xenial excluded:

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_12:24:43-rados-wip-hammer-backports-distro-basic-smithi/

Re-pushed new integration branch with https://github.com/ceph/ceph/pull/12743 and running the full 112 jobs again:

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_15:48:30-rados-wip-hammer-backports-distro-basic-smithi/ (four fail, three dead)

Actions
Copy link
 #30

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_15:51:41-rgw-wip-hammer-backports-distro-basic-smithi/

Re-running failed tests with --suite-repo https://github.com/ceph/ceph.git --suite-branch hammer (to avoid issue#18384):

./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --suite-repo https://github.com/ceph/ceph.git --suite-branch hammer --filter="$filter" qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-02_19:50:30-rgw-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #31

Updated by Nathan Cutler over 7 years ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:38:11-rbd-wip-hammer-backports-distro-basic-smithi/ (two fail, one dead)

Re-running dead and failed jobs with fix for issue#18388

teuthology-suite --priority 101 --suite rbd --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="$filter" qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:52:46-rbd-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #32

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:39:27-fs-wip-hammer-backports-distro-basic-smithi/ (one dead job)

Actions
Copy link
 #33

Updated by Nathan Cutler over 7 years ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz

fail http://pulpito.ceph.com:80/smithfarm-2017-01-02_16:40:28-powercycle-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #34

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #35

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:23:21-rgw-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #36

Updated by Nathan Cutler over 7 years ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

running http://pulpito.ceph.com:80/smithfarm-2017-01-03_08:24:28-powercycle-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #37

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #38

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-03_20:59:45-rados-wip-hammer-backports-distro-basic-smithi/

Re-running two failed (centos-only) jobs without "qa/distros/all/ubuntu_14.04.yaml":

one pass, one stuck in "queued" state http://pulpito.ceph.com:80/smithfarm-2017-01-04_19:05:09-rados-wip-hammer-backports-distro-basic-smithi/

Re-running one failed job with "qa/distros/all/ubuntu_14.04.yaml":

pass http://pulpito.ceph.com:80/smithfarm-2017-01-04_19:06:53-rados-wip-hammer-backports-distro-basic-smithi/

Re-running one job that seemed to get stuck in "Queued" state:

pass http://pulpito.ceph.com:80/smithfarm-2017-01-04_20:52:15-rados-wip-hammer-backports-distro-basic-smithi/

Re-running last job that got stuck in "Queued" state:

pass http://pulpito.ceph.com:80/smithfarm-2017-01-05_10:54:32-rados-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #39

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:01:35-rgw-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #40

Updated by Nathan Cutler over 7 years ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:02:28-rbd-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #41

Updated by Nathan Cutler over 7 years ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:09:18-powercycle-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #42

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-03_21:09:50-fs-wip-hammer-backports-distro-basic-smithi/

Though technically "fail", this run is actually "pass" because the only failure can safely be ignored.

Actions
Copy link
 #43

Updated by Nathan Cutler over 7 years ago

Actions
Copy link
 #44

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-05_20:47:44-rgw-wip-hammer-backports-distro-basic-smithi/ (one failed job)

Re-running the failed job:

teuthology-suite --priority 101 --suite rgw --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/btrfs.yaml overrides.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}" --dry-run qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-06_16:38:29-rgw-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #46

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-13_18:40:44-rados-wip-hammer-backports-distro-basic-smithi/

  • failed jobs - not expected to succeed (?) but fix might be feasible, see http://tracker.ceph.com/issues/18069#note-7
    • rados/singleton-nomsgr/{all/11429.yaml}
    • rados/singleton-nomsgr/{all/13234.yaml}
  • plus three tests that need to be re-run without "qa/distros/all/ubuntu_14.04.yaml" because teuthology thinks they should be run on "centos 14.04"

Re-running the latter three failed tests without qa/distros/all/ubuntu_14.04.yaml

./virtualenv/bin/teuthology-suite --priority 101 --suite rados --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro --filter="$filter"

pass http://pulpito.ceph.com:80/smithfarm-2017-01-20_20:54:24-rados-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #47

Updated by Nathan Cutler over 7 years ago

powercycle

./virtualenv/bin/teuthology-suite -l2 -v -c wip-hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-13_18:43:12-powercycle-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #48

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #49

Updated by Nathan Cutler over 7 years ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

pass http://pulpito.ceph.com:80/smithfarm-2017-01-20_23:47:14-rbd-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #50

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-20_23:49:04-fs-wip-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #51

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #52

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #53

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #54

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #55

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..wip-hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #56

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --email ncutler@suse.cz --ceph wip-hammer-backports --machine-type smithi -k distro qa/distros/all/ubuntu_14.04.yaml

fail http://pulpito.ceph.com:80/smithfarm-2017-01-21_15:59:00-fs-wip-hammer-backports-distro-basic-smithi/

  • The only failure is fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml} which was not expected to pass

Ruled a pass

Actions
Copy link
 #57

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #58

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #59

Updated by Nathan Cutler about 7 years ago

  • Description updated (diff)
Actions
Copy link
 #60

Updated by Yuri Weinstein about 7 years ago

  • Description updated (diff)
Actions
Copy link
 #61

Updated by Yuri Weinstein about 7 years ago

QE VALIDATION (STARTED 1/23/17)

re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v0.94.10_QE_validation_notes
COMMAND LINE => CEPH_BRANCH=83af8cdaaa6d94404e6146b68e532a784e3cc99c; MACHINE_NAME=vps; teuthology-suite -v -S $CEPH_BRANCH -m $MACHINE_NAME -k distro -s rados -e $CEPH_QA_EMAIL --suite-branch hammer

(Note: PASSED / FAILED - indicates "TEST IS IN PROGRESS")

All suites won't test on ubuntu 16.04 http://tracker.ceph.com/issues/18382

The following tests are not expected to pass:

  • fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml} -> http://tracker.ceph.com/issues/14716
  • rados/singleton-nomsgr/{all/11429.yaml}
  • rados/singleton-nomsgr/{all/13234.yaml}
Suite Runs/Reruns Notes/Issues
rados ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_16:20:34-rados-master-distro-basic-vps/ FAILED -subset $(expr $RANDOM % 20)/20
http://pulpito.ceph.com/teuthology-2017-01-26_17:48:56-rados-hammer---basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-28_16:54:09-rados-hammer---basic-smithi/ approved by Josh
rados centos http://pulpito.ceph.com/teuthology-2017-01-25_16:32:25-rados-master-distro-basic-vps/ approved by Josh
http://pulpito.ceph.com/teuthology-2017-01-26_17:52:58-rados-hammer---basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-28_16:55:30-rados-hammer---basic-smithi/ approved by Josh
rbd ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_22:00:24-rbd-master-distro-basic-smithi/ ubuntu PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_17:28:55-rbd-hammer---basic-mira/
rbd centos http://pulpito.ceph.com/teuthology-2017-01-25_22:04:27-rbd-master-distro-basic-smithi/ centos PASSED
rgw ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_22:18:36-rgw-master-distro-basic-smithi/ ubuntu PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_17:22:10-rgw-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-28_16:26:40-rgw-hammer---basic-smithi/
rgw centos http://pulpito.ceph.com/teuthology-2017-01-25_22:21:30-rgw-master-distro-basic-smithi/ centos FAILED #18735
http://pulpito.ceph.com/teuthology-2017-01-26_17:19:58-rgw-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-28_16:28:47-rgw-hammer---basic-smithi/ Approval/decision by Yehuda/Sage needed
http://pulpito.ceph.com/teuthology-2017-01-28_22:35:40-rgw-hammer---basic-vps/ trying centos 7.2
http://pulpito.ceph.com/teuthology-2017-02-14_16:21:46-rgw-hammer-distro-basic-smithi/
 retesting with fix http://tracker.ceph.com/issues/18744
fs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_18:02:24-fs-master-distro-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by John
"The MDL_WriteError one is the bug from when ENOSPC stuff was
backported wrongly, the FileJournal/FileStore assertions are from btrfs."
http://pulpito.ceph.com/teuthology-2017-01-28_16:31:30-fs-hammer---basic-smithi/
http://pulpito.ceph.com/teuthology-2017-01-31_17:18:26-fs-hammer-distro-basic-smithi/ run off gitbuilders
fs centos http://pulpito.ceph.com/teuthology-2017-01-26_18:03:38-fs-master-distro-basic-smithi/ FAILED noise? approved by John
http://pulpito.ceph.com/teuthology-2017-01-28_16:32:30-fs-hammer---basic-smithi/
krbd ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_20:58:56-krbd-master-testing-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by Ilya
http://pulpito.ceph.com/teuthology-2017-01-28_16:38:13-krbd-hammer-testing-basic-smithi/
krbd centos http://pulpito.ceph.com/teuthology-2017-01-26_21:01:15-krbd-master-testing-basic-smithi/ FAILED looks like the same in v0.94.8 http://tracker.ceph.com/issues/15895#note-33 approved by Ilya
http://pulpito.ceph.com/teuthology-2017-01-28_16:38:58-krbd-hammer-testing-basic-smithi/
kcephfs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:03:50-kcephfs-master-testing-basic-smithi/
 PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:40:15-kcephfs-hammer-testing-basic-smithi/
kcephfs centos http://pulpito.ceph.com/teuthology-2017-01-26_21:05:21-kcephfs-master-testing-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:41:05-kcephfs-hammer-testing-basic-smithi/
knfs ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:08:14-knfs-master-testing-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:42:08-knfs-hammer-testing-basic-smithi/
knfs centos http://pulpito.ceph.com/teuthology-2017-01-26_21:08:46-knfs-master-testing-basic-smithi/ PASSED
hadoop ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:09:55-hadoop-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:43:53-hadoop-hammer---basic-smithi/
hadoop centos http://pulpito.ceph.com/teuthology-2017-01-26_21:10:40-hadoop-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:45:27-hadoop-hammer---basic-smithi/
samba ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:11:17-samba-master-distro-basic-smithi/ FAILED shaman build issues approved by Sage
http://pulpito.ceph.com/teuthology-2017-01-30_18:15:33-samba-hammer-distro-basic-smithi/ run off gitbuilders, packaging issues
samba centos http://pulpito.ceph.com/teuthology-2017-01-26_21:11:58-samba-master-distro-basic-smithi/ FAILED shaman build issues approved by Sage
rest ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:12:36-rest-master-distro-basic-smithi/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-28_16:47:24-rest-hammer---basic-smithi/
rest centos http://pulpito.ceph.com/teuthology-2017-01-26_21:14:18-rest-master-distro-basic-smithi/ PASSED
ceph-deploy ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:15:14-ceph-deploy-master-distro-basic-vps/ FAILED #18737, #18736
http://pulpito.ceph.com/teuthology-2017-01-30_17:38:19-ceph-deploy-hammer-distro-basic-vps/
ceph-deploy centos http://pulpito.ceph.com/teuthology-2017-01-26_21:15:47-ceph-deploy-master-distro-basic-vps/ FAILED same as 0.94.08 see #15895
upgrade/client-upgrade ubuntu http://pulpito.ceph.com/teuthology-2017-01-26_21:29:02-upgrade:client-upgrade-master-distro-basic-vps/
upgrade/client-upgrade centos http://pulpito.ceph.com/teuthology-2017-01-26_21:30:09-upgrade:client-upgrade-master-distro-basic-vps/
http://pulpito.ceph.com/teuthology-2017-01-30_20:48:08-upgrade:client-upgrade-hammer-distro-basic-vps/ run off gitbuilders a must! PASSED
upgrade/firefly-x (hammer) http://pulpito.ceph.com/teuthology-2017-01-30_20:52:09-upgrade:firefly-x-hammer-distro-basic-vps/ FAILED run off gitbuilders, approved by Sage
http://pulpito.ceph.com/teuthology-2017-01-31_16:19:21-upgrade:firefly-x-hammer---basic-smithi/
upgrade/hammer http://pulpito.ceph.com/teuthology-2017-01-26_21:34:09-upgrade:hammer-master-distro-basic-vps/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-30_20:57:57-upgrade:hammer-hammer-distro-basic-vps/ run off gitbuilders
http://pulpito.ceph.com/teuthology-2017-01-31_16:18:48-upgrade:hammer-hammer---basic-smithi/ 2 failed b/c of noise
upgrade/hammer-x (jewel) http://pulpito.ceph.com/teuthology-2017-02-07_18:15:23-upgrade:hammer-x-jewel-distro-basic-vps/ PASSED
http://pulpito.ceph.com/yuriw-2017-02-08_17:22:47-upgrade:hammer-x-jewel---basic-smithi/ run off gitbuilders
powercycle ubuntu http://pulpito.ceph.com/teuthology-2017-01-25_16:42:38-powercycle-master-testing-basic-mira/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_16:47:48-powercycle-hammer---basic-mira/
powercycle centos http://pulpito.ceph.com/teuthology-2017-01-25_16:43:42-powercycle-master-testing-basic-mira/ PASSED
http://pulpito.ceph.com/teuthology-2017-01-26_16:58:05-powercycle-hammer---basic-mira/
http://pulpito.ceph.com/teuthology-2017-01-28_16:50:25-powercycle-hammer-testing-basic-smithi/
(DON'T REMOVE BELOW)
Suite Runs/Reruns Notes/Issues
PASSED / FAILED

centos

Actions
Copy link
 #62

Updated by Nathan Cutler about 7 years ago

rgw valgrind issues bisect

ubuntu ./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph hammer --machine-type smithi --ceph-repo http://github.com/ceph/ceph.git --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" qa/distros/all/ubuntu_14.04.yaml

centos 7.3 ./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph hammer --machine-type smithi --ceph-repo http://github.com/ceph/ceph.git --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" ~/os_centos.yaml

Tip of hammer ("bad" version for bisect) 83af8cdaaa6d94404e6146b68e532a784e3cc99c

Ubuntu 14.04

CentOS 7.3

0.94.9 (hopefully "good" version for bisect) wip-hammer-0-94-9

./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rgw/verify --email ncutler@suse.com --ceph wip-hammer-0-94-9 --machine-type smithi --ceph-repo http://github.com/ceph/ceph-ci.git --suite-branch hammer --suite-repo http://github.com/ceph/ceph.git --filter="valgrind" ~/os_centos.yaml

Since the problem is present in the 0.94.9 release, we can be sure it's not due to a regression introduced in 0.94.10.

Actions
Copy link
 #63

Updated by Nathan Cutler about 7 years ago

  • Description updated (diff)
Actions
Copy link
 #64

Updated by Nathan Cutler about 7 years ago 

(04:03:19 PM) sage: smithfarm: i think we can ignore hte notcmalloc hammer stuff and move on...
(04:03:59 PM) smithfarm: sage: ok, so afaict that means hammer 0.94.10 has passed qE
(04:04:33 PM) smithfarm: unless you think we should fix upgrade/firefly-x
(04:04:36 PM) sage: yeah i think so
(04:04:37 PM) sage: nope :)
Actions
Copy link
 #65

Updated by Abhishek Lekshmanan about 7 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF