Project

General

Profile

Actions
Copy link

Tasks #15895

closed

hammer v0.94.8

Added by Loïc Dachary almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Nathan Cutler
Target version:
Ceph - v0.94.8
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

Workflow

  • Preparing the release
  • Cutting the release
    • Nathan asks Sage if a point release should be published YES
    • Nathan gets approval from all leads
      • Yehuda, rgw - YES
      • John, CephFS - YES
      • Jason, RBD - YES
      • Sam, rados - YES
    • Nathan prepares draft release notes - DONE https://github.com/ceph/ceph/pull/10730
    • Sage writes and commits the release notes - DONE https://github.com/ceph/ceph/pull/10730
    • Nathan informs Yuri that the branch is ready for testing - DONE 2016-08-15
    • Yuri runs additional integration tests - DONE 2016-08-25
    • If Yuri discovers new bugs that need to be backported urgently (i.e. their priority is set to Urgent), the release goes back to being prepared, it was not ready after all - NO BLOCKERS FOUND
    • Yuri informs Alfredo that the branch is ready for release - DONE 2016-08-25
    • Alfredo creates the packages and sets the release tag - DONE

Release information

Actions
Copy link
 #1

Updated by Nathan Cutler almost 8 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #2

Updated by Nathan Cutler almost 8 years ago

RBD

teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=3/5

fail http://pulpito.ceph.com/smithfarm-2016-06-02_00:35:31-rbd-hammer-backports---basic-smithi/

Re-scheduled failed job:

pass http://pulpito.ceph.com/smithfarm-2016-06-15_06:49:57-rbd-hammer-backports---basic-smithi/

  • rbd/singleton/{all/qemu-iotests-no-cache.yaml} Command failed (workunit test rbd/qemu-iotests.sh) on smithi046 with status 1
Actions
Copy link
 #3

Updated by Nathan Cutler almost 8 years ago

rados

teuthology-suite --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=712/2000

red http://pulpito.ceph.com/smithfarm-2016-06-02_00:38:50-rados-hammer-backports---basic-smithi/

Actions
Copy link
 #4

Updated by Nathan Cutler almost 8 years ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=1/5

red http://pulpito.ceph.com/smithfarm-2016-06-02_00:42:57-rgw-hammer-backports---basic-smithi/

Rescheduled failed jobs:

red http://pulpito.ceph.com/smithfarm-2016-06-15_06:42:48-rgw-hammer-backports---basic-smithi/

(same array of bugs)

Actions
Copy link
 #5

Updated by Nathan Cutler almost 8 years ago

powercycle

teuthology-suite -l2 -v -c hammer-backports -k testing -m smithi -s powercycle -p 1000 --email ncutler@suse.cz

running http://pulpito.ceph.com/smithfarm-2016-06-02_00:45:29-powercycle-hammer-backports-testing-basic-smithi/

Actions
Copy link
 #6

Updated by Nathan Cutler almost 8 years ago

fs

teuthology-suite --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=3/5

running http://pulpito.ceph.com/smithfarm-2016-06-02_00:47:31-fs-hammer-backports---basic-smithi/

Actions
Copy link
 #7

Updated by Nathan Cutler almost 8 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #8

Updated by Nathan Cutler almost 8 years ago

rados

teuthology-suite --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=1342/2000

red http://pulpito.ceph.com/smithfarm-2016-07-02_01:06:27-rados-hammer-backports---basic-smithi/

(Most, if not all, of the failures have "ceph-osd: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr<T>::operator->() const [with T = Connection]: Assertion `px != 0' failed." somewhere in the teuthology log)

Also, https://github.com/ceph/ceph/pull/8077 is causing an extra '\n' to be written after every line in the OSD log.

Rescheduled dead and failed jobs:

red http://pulpito.ceph.com/smithfarm-2016-07-08_14:05:08-rados-hammer-backports---basic-smithi/
Actions
Copy link
 #9

Updated by Nathan Cutler almost 8 years ago

rgw

teuthology-suite --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=0/5

red http://pulpito.ceph.com/smithfarm-2016-07-02_01:12:45-rgw-hammer-backports---basic-smithi/

Re-running the four failed jobs:

red http://pulpito.ceph.com/smithfarm-2016-07-08_12:39:54-rgw-hammer-backports---basic-smithi/
Actions
Copy link
 #10

Updated by Nathan Cutler almost 8 years ago

powercycle

teuthology-suite -l2 -v -c hammer-backports -k distro -m smithi -s powercycle -p 1000 --email ncutler@suse.cz red http://pulpito.ceph.com/smithfarm-2016-07-02_01:16:52-powercycle-hammer-backports-distro-basic-smithi/
  • one successful job, one dead job
Actions
Copy link
 #11

Updated by Nathan Cutler almost 8 years ago

fs

teuthology-suite --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=1/5

red http://pulpito.ceph.com/smithfarm-2016-07-02_01:22:28-fs-hammer-backports---basic-smithi/

Re-scheduled dead and failed jobs:

red http://pulpito.ceph.com/smithfarm-2016-07-08_15:27:38-fs-hammer-backports---basic-smithi/
Actions
Copy link
 #12

Updated by Nathan Cutler almost 8 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #13

Updated by Nathan Cutler over 7 years ago

rados

teuthology-suite --priority 100 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=829/2000

fail http://pulpito.ceph.com/smithfarm-2016-07-20_00:22:41-rados-hammer-backports---basic-smithi/

Re-scheduled failed and dead jobs using ceph-qa-suite branch from https://github.com/ceph/ceph-qa-suite/pull/1089:

./virtualenv/bin/teuthology-suite --filter="$filter" --priority 101 --suite rados --suite-branch wip-832-hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

running http://pulpito.ceph.com/smithfarm-2016-07-21_14:08:16-rados-hammer-backports---basic-smithi/

Since the Sepia lab is not working this weekend, I schedule a rados run in OVH:

cw -v ceph-qa-suite --simultaneous-jobs 20 --suite rados --subset $(expr $RANDOM % 2000)/2000 --ceph-qa-suite-git-url http://github.com/smithfarm/ceph-qa-suite --suite-branch wip-hammer-openstack-fixes --ceph hammer-backports distros/all/centos_7.2.yaml

The "wip-hammer-openstack-fixes" ceph-qa-suite branch is just hammer with an OpenStack-specific fix and the patches from https://github.com/ceph/ceph-qa-suite/pull/1089

running http://158.69.66.4:8081/ubuntu-2016-07-23_16:08:12-rados-hammer-backports---basic-openstack/
Actions
Copy link
 #14

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=1/5

fail http://pulpito.ceph.com/smithfarm-2016-07-20_00:29:35-rgw-hammer-backports---basic-smithi/

Actions
Copy link
 #15

Updated by Nathan Cutler over 7 years ago

powercycle

teuthology-suite -l2 -v -c hammer-backports -k distro -m smithi -s powercycle -p 1000 --email ncutler@suse.cz

fail http://pulpito.ceph.com/smithfarm-2016-07-20_00:33:39-powercycle-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #16

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 100 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=1/5

fail http://pulpito.ceph.com/smithfarm-2016-07-20_00:50:13-fs-hammer-backports---basic-smithi/

Actions
Copy link
 #17

Updated by Nathan Cutler over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/hammer..hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #18

Updated by Nathan Cutler over 7 years ago

rados

CentOS 7.2 run:

cw -v ceph-qa-suite --simultaneous-jobs 20 --suite rados --subset $(expr $RANDOM % 2000)/2000 --ceph-qa-suite-git-url http://github.com/smithfarm/ceph-qa-suite --suite-branch wip-hammer-openstack-fixes --ceph hammer-backports distros/all/centos_7.2.yaml

fail http://158.69.66.4:8081/ubuntu-2016-07-24_20:00:22-rados-hammer-backports---basic-openstack/

Re-scheduled dead/failed jobs:

teuthology-openstack --filter="$filter" --suite=rados --ceph-qa-suite-git-url http://github.com/smithfarm/ceph-qa-suite --suite-branch wip-hammer-openstack-fixes --ceph hammer-backports distros/all/centos_7.2.yaml

fail http://158.69.66.4:8081/ubuntu-2016-07-25_07:08:00-rados-hammer-backports---basic-openstack/

Ubuntu 14.04 run:

cw -v ceph-qa-suite --simultaneous-jobs 20 --suite rados --subset $(expr $RANDOM % 2000)/2000 --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-hammer --ceph hammer-backports distros/all/ubuntu_14.04.yaml

running http://158.69.78.47:8081/ubuntu-2016-07-27_15:27:10-rados-hammer-backports---basic-openstack/

Actions
Copy link
 #19

Updated by Nathan Cutler over 7 years ago

rgw

teuthology-suite --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=2/5

fail http://pulpito.ceph.com/smithfarm-2016-07-25_05:04:48-rgw-hammer-backports---basic-smithi/

Re-running dead and failed jobs:

fail http://pulpito.ceph.com/smithfarm-2016-07-27_09:14:19-rgw-hammer-backports---basic-smithi/

Re-running valgrind failures to verify reproducibility:

fail http://pulpito.ceph.com/smithfarm-2016-07-28_06:29:58-rgw-hammer-backports---basic-smithi/
  • three valgrind failures, but they are unrelated to RGW
Actions
Copy link
 #20

Updated by Nathan Cutler over 7 years ago

powercycle

teuthology-suite -l2 -v -c hammer-backports -k distro -m smithi -s powercycle -p 101 --email ncutler@suse.cz

fail http://pulpito.ceph.com/smithfarm-2016-07-25_05:08:03-powercycle-hammer-backports-distro-basic-smithi/

Re-running failed job:

pass http://pulpito.ceph.com/smithfarm-2016-07-27_09:16:35-powercycle-hammer-backports-distro-basic-smithi/

Actions
Copy link
 #21

Updated by Nathan Cutler over 7 years ago

fs

teuthology-suite --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --email ncutler@suse.cz --ceph hammer-backports --machine-type smithi

Subset=0/5

fail http://pulpito.ceph.com/smithfarm-2016-07-25_05:09:12-fs-hammer-backports---basic-smithi/

Re-running dead and failed jobs:

fail http://pulpito.ceph.com/smithfarm-2016-07-27_09:18:04-fs-hammer-backports---basic-smithi/
Actions
Copy link
 #22

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #23

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #24

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #25

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #26

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #27

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #28

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #29

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #30

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #31

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #32

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #33

Updated by Yuri Weinstein over 7 years ago

QE VALIDATION (STARTED 8/15/16)

(Note: PASSED / FAILED - indicates "TEST IS IN PROGRESS")

re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v0.94.8_QE_validation_notes
command line CEPH_BRANCH=08277b7bc7c0e533c3fd56a0040dc0ddc74637d6; MACHINE_NAME=vps; teuthology-suite -v -S $CEPH_BRANCH -m $MACHINE_NAME -k distro -s rados -e $CEPH_QA_EMAIL --suite-branch hammer

Suite Runs/Reruns Notes/Issues
rados http://pulpito.ceph.com/yuriw-2016-08-15_16:27:27-rados-master-distro-basic-vps/ PASSED
-subset $(expr $RANDOM % 20)/20, #17035
http://pulpito.ceph.com/yuriw-2016-08-16_16:41:19-rados-master-distro-basic-vps/ failed job due to memory on vps
rbd http://pulpito.ceph.com/yuriw-2016-08-15_16:34:54-rbd-master-distro-basic-vps/ PASSED
#17049, #17048
#17035, #17045,#17046
http://pulpito.ceph.com/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/ "qemu on smithi with --filter qemu and the rest on vps with --filter-out qemu"
http://pulpito.ceph.com/yuriw-2016-08-16_20:43:06-rbd-master-distro-basic-smithi/ on smithi
rgw http://pulpito.ceph.com/yuriw-2016-08-15_16:43:26-rgw-master-distro-basic-smithi/ PASSED
#17035
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_16:24:35-rgw-master-distro-basic-smithi/
fs http://pulpito.ceph.com/yuriw-2016-08-15_17:41:47-fs-master---basic-smithi/ PASSED #17035
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_16:28:07-fs-master---basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_16:28:59-fs-master---basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-18_20:11:00-fs-master---basic-smithi/ FAILED #14716 same as in v0.94.7 see #14692
krbd http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:37:13-krbd-master-testing-basic-smithi/ FAILED #17083 Approved by Ilya
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_16:31:36-krbd-master-testing-basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-18_20:13:38-krbd-master-testing-basic-smithi/
kcephfs http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:38:36-kcephfs-master-testing-basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_16:33:33-kcephfs- Suite Runs/Reruns Notes/Issues
PASSED / FAILED
master-testing-basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-18_20:16:32-kcephfs-master-testing-basic-smithi/ PASSED
knfs http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:39:23-knfs-master-testing-basic-smithi/ FAILED Approved by Greg
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_20:57:47-knfs-master-testing-basic-smithi/ #17074
hadoop http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:40:40-hadoop-master---basic-smithi/ PASSED
samba http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:41:22-samba-master---basic-smithi/ FAILED #14714 same as in v0.94.6 see #13356
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_20:55:06-samba-master---basic-smithi/
rest http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:41:53-rest-master---basic-smithi/ PASSED
ceph-deploy http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:43:23-ceph-deploy-master-distro-basic-vps/ FAILED #14223
same as in v0.94.7 see #13356
upgrade/client-upgrade http://pulpito.ceph.com/yuriw-2016-08-16_23:52:01-upgrade:client-upgrade-master-distro-basic-vps/ PASSED
upgrade/dumpling-firefly-x DEPRECATED, OKed by Sage
upgrade/firefly-x (hammer) http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_21:59:58-upgrade:firefly-x-master-distro-basic-mira/ FAILED #17075 DEPRECATED, OKed by Sage
http://pulpito.ceph.com/yuriw-2016-08-25_15:28:54-upgrade:firefly-x-hammer-distro-basic-vps/ re-run with Nathan's fix and PASSED
upgrade/hammer http://pulpito.ceph.com/yuriw-2016-08-16_22:00:41-upgrade:hammer-master-distro-basic-vps/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-17_16:46:08-upgrade:hammer-master-distro-basic-vps/ PASSED #17075
failed jobs due to giant step, removed now https://github.com/ceph/ceph-qa-suite/pull/1125
upgrade/hammer-x (infernalis) http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_22:13:31-upgrade:hammer-x-infernalis-distro-basic-vps/ DEPRECATED OKed by Sage
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-18_20:36:06-upgrade:hammer-x-infernalis-distro-basic-vps/
upgrade/firefly-hammer-x (infernalis) http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-16_22:15:12-upgrade:firefly-hammer-x-infernalis-distro-basic-mira/ need rerun #17075 DEPRECATED, OKed by Sage
upgrade/hammer-x (jewel) http://pulpito.ceph.com/teuthology-2016-08-11_18:15:02-upgrade:hammer-x-jewel-distro-basic-vps/ PASSED
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-19_16:42:44-upgrade:hammer-x-jewel-distro-basic-smithi/ FAILED #17075
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-08-23_16:10:30-upgrade:hammer-x-jewel-distro-basic-smithi/
http://pulpito.ceph.com/yuriw-2016-08-23_22:40:44-upgrade:hammer-x-jewel-distro-basic-smithi/
ceph-disk #17129 ???
powercycle http://pulpito.ceph.com/yuriw-2016-08-16_16:29:15-powercycle-master-testing-basic-mira/ PASSED
(DON'T REMOVE BELOW)
Suite Runs/Reruns Notes/Issues
PASSED / FAILED
Actions
Copy link
 #34

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #35

Updated by Yuri Weinstein over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #36

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #37

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #38

Updated by Nathan Cutler over 7 years ago

Advanced the SHA1 from 08277b7 to eb27865 ("latest hammer branch") as agreed on IRC:

(10:40:50 PM) sage: did any tests cover the new sha1?
(10:40:52 PM) yuriw: can I use two ceph dirs on the same node and run make, cmake, tests etc. ?
(10:41:10 PM) yuriw: sjusthm ^
(10:41:16 PM) sage: oh, it's just telling rocksdb not to link against tcmalloc when passed the weird options for the -notcmalloc builder.
(10:41:31 PM) sage: alfredodeza: so ok to build latest hammer branch.
(10:41:36 PM) alfredodeza: ok
(10:42:00 PM) sage: yuriw: yeah, just use different checkouts

Actions
Copy link
 #39

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF