Project

General

Profile

Actions
Copy link

Tasks #17487

closed

jewel v10.2.4

Added by Loïc Dachary over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Abhishek Varshney
Target version:
Ceph - v10.2.4
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

Workflow

  • Preparing the release
  • Cutting the release
    • Abhishek V. asks Abhishek L. if a point release should be published YES
    • Abhishek V. gets approval from all leads
      • Yehuda, rgw YES
      • John, CephFS YES
      • Jason, RBD YES
      • Sam, rados YES
    • Abhishek L. writes and commits the release notes, including: 
      • Set require_jewel_osds osdmap flag on upgrade from hammer

When the last hammer OSD in a cluster containing jewel MONs is 
upgraded to jewel, as of 10.2.4 the jewel MONs will issue this 
warning: "all OSDs are running jewel or later but the 
'require_jewel_osds' osdmap flag is not set" and change the 
cluster health status to HEALTH_WARN.

This is a signal for the admin to do "ceph osd set 
require_jewel_osds" - by doing this, the admin acknowledges that 
there is no downgrade path.
    • Abhishek V. informs Yuri that the branch is ready for testing DONE
    • Yuri runs additional integration tests IN PROGRESS
    • If Yuri discovers new bugs that need to be backported urgently (i.e. their priority is set to Urgent), the release goes back to being prepared, it was not ready after all
    • Yuri informs Alfredo that the branch is ready for release
    • Alfredo creates the packages and sets the release tag

Release information

Actions
Copy link
 #1

Updated by Loïc Dachary over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #2

Updated by Loïc Dachary over 7 years ago

rbd

teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=2/5

Re-running failed jobs

Actions
Copy link
 #3

Updated by Loïc Dachary over 7 years ago

rgw

teuthology-suite -k distro --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=1/5

Re-running the failed tests without https://github.com/ceph/ceph/pull/10832 to verify it is responsible for them

Actions
Copy link
 #4

Updated by Loïc Dachary over 7 years ago

rados

teuthology-suite -k distro --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=1846/2000

Re-running failed jobs

Actions
Copy link
 #5

Updated by Loïc Dachary over 7 years ago

fs

teuthology-suite -k distro --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

INFO:teuthology.suite:Passed subset=3/5

Re-running failed tests

Actions
Copy link
 #6

Updated by Loïc Dachary over 7 years ago

powercycle

teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
Actions
Copy link
 #7

Updated by Loïc Dachary over 7 years ago

Upgrade

teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 1000 machine_types/vps.yaml teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 1000 machine_types/vps.yaml
Actions
Copy link
 #8

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #9

Updated by Loïc Dachary over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #10

Updated by Loïc Dachary over 7 years ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-run failed tests

The dead jobs are because of a known bug rbd-nbd IO hang

Actions
Copy link
 #11

Updated by Loïc Dachary over 7 years ago

rgw

teuthology-suite -k distro --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-running failed tests

Actions
Copy link
 #12

Updated by Loïc Dachary over 7 years ago

rados

teuthology-suite -k distro --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-run failed tests

Actions
Copy link
 #13

Updated by Loïc Dachary over 7 years ago

fs

teuthology-suite -k distro --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-run failed tests

Actions
Copy link
 #14

Updated by Loïc Dachary over 7 years ago

powercycle

teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
Actions
Copy link
 #15

Updated by Loïc Dachary over 7 years ago

Upgrade

teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml

Re-run dead jobs

teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml

Re-run failed jobs

Actions
Copy link
 #16

Updated by Loïc Dachary over 7 years ago

git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
Actions
Copy link
 #17

Updated by Loïc Dachary over 7 years ago

rbd

teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-running failed jobs

The dead jobs are because of a known bug rbd-nbd IO hang

Actions
Copy link
 #18

Updated by Loïc Dachary over 7 years ago

rgw

teuthology-suite -k distro --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-running failed jobs

Actions
Copy link
 #19

Updated by Loïc Dachary over 7 years ago

rados

teuthology-suite -k distro --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-running failed jobs

Running rados/singleton/{rados.yaml all/cephtool.yaml fs/xfs.yaml msgr/random.yaml msgr-failures/many.yaml} against jewel to check if it shows

Running rados/singleton/{rados.yaml all/cephtool.yaml fs/xfs.yaml msgr/random.yaml msgr-failures/many.yaml} against jewel-backports 10 times to see how frequently it shows up

Actions
Copy link
 #20

Updated by Loïc Dachary over 7 years ago

fs

teuthology-suite -k distro --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi

Re-running failed jobs

Actions
Copy link
 #21

Updated by Loïc Dachary over 7 years ago

powercycle

teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
Actions
Copy link
 #22

Updated by Loïc Dachary over 7 years ago

Upgrade

teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml

Re-running failed jobs

teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml

Re-running failed jobs

Actions
Copy link
 #23

Updated by Loïc Dachary over 7 years ago

ceph-disk

teuthology-suite -k distro --verbose --suite ceph-disk --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
Actions
Copy link
 #24

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #25

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #26

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #27

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #28

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #29

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #30

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #31

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #32

Updated by Yuri Weinstein over 7 years ago

QE VALIDATION (STARTED 11/7/16)

(Note: PASSED / FAILED - indicates "TEST IS IN PROGRESS")

re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v10.2.4_QE_validation_notes

command line CEPH_BRANCH=jewel; MACHINE_NAME=vps; teuthology-suite -v -S $CEPH_BRANCH -m $MACHINE_NAME -k distro -s rados -e $CEPH_QA_EMAIL --suite-branch jewel

Suite Runs/Reruns Notes/Issues
rados http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_16:39:57-rados-jewel-distro-basic-smithi/ PASSED
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:18:07-rados-jewel-distro-basic-smithi/
rgw http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-10_15:57:16-rgw-jewel-distro-basic-smithi/ FAILED #16129 #18015
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-10_18:41:26-rgw-jewel-distro-basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-10_21:17:57-rgw-jewel-distro-basic-smithi/ after pushed https://github.com/ceph/ceph-qa-suite/pull/1136
rbd http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_16:46:17-rbd-jewel-distro-basic-smithi/ PASSED
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:20:03-rbd-jewel-distro-basic-smithi/
fs http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_16:52:19-fs-jewel-distro-basic-smithi/ FAILED #17832 Approved by John
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:23:29-fs-jewel-distro-basic-smithi/
krbd http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:04:25-krbd-jewel-testing-basic-smithi/ FAILED #17221 Approved by Ilya
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:25:31-krbd-jewel-distro-basic-smithi/
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_19:33:14-krbd-jewel-testing-basic-smithi/ on -k testing
kcephfs http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:06:21-kcephfs-jewel-testing-basic-smithi/ PASSED
knfs http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:08:28-knfs-jewel-testing-basic-smithi/ FAILED #16397 same as in 10.2.3
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-09_01:10:20-knfs-jewel-testing-basic-smithi/
rest http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:09:43-rest-jewel-distro-basic-smithi/ PASSED
hadoop http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:10:15-hadoop-jewel-distro-basic-smithi/ PASSED
samba http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:10:58-samba-jewel-distro-basic-smithi/ PASSED
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:30:02-samba-jewel-distro-basic-smithi/
ceph-deploy http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:12:08-ceph-deploy-jewel-distro-basic-vps/ PASSED #17352, #17849, #17816
http://pulpito.ceph.com/teuthology-2016-11-10_10:55:01-ceph-deploy-jewel-distro-basic-vps/
ceph-disk http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:12:50-ceph-disk-jewel-distro-basic-vps/ PASSED
upgrade/client-upgrade http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:33:02-upgrade:client-upgrade-jewel-distro-basic-smithi PASSED
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-09_01:13:23-upgrade:client-upgrade-jewel-distro-basic-smithi/
upgrade/hammer-x (jewel) http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-08_16:36:26-upgrade:hammer-x-jewel-distro-basic-vps/ FAILED #17847
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-09_15:03:11-upgrade:hammer-x-jewel-distro-basic-vps/ --suite-branch wip-17734-jewel
http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-14_16:55:44-upgrade:hammer-x-jewel-distro-basic-vps/
upgrade/infernalis-x (jewel) http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-14_16:21:41-upgrade:infernalis-x-jewel-distro-basic-vps/ UNSUPPORTED #17683, #17874
upgrade/jewel-x/point-to-point-x http://pulpito.ceph.com/yuriw-2016-11-16_15:07:00-upgrade:jewel-x:point-to-point-x-jewel-distro-basic-smithi/ PASSED
powercycle http://pulpito.ceph.com/yuriw-2016-11-11_15:40:06-powercycle-jewel-distro-basic-smithi/ PASSED after https://github.com/ceph/ceph-qa-suite/pull/1246
http://pulpito.ceph.com/yuriw-2016-11-11_17:26:34-powercycle-jewel-distro-basic-smithi/
http://pulpito.ceph.com/yuriw-2016-11-12_15:57:06-powercycle-jewel-distro-basic-smithi/
PASSED / FAILED
Actions
Copy link
 #33

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #34

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #35

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #36

Updated by Nathan Cutler over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #37

Updated by Abhishek Lekshmanan over 7 years ago

Running a suite for the last 2 prs which will make it to 10.2.4

This is the current jewel branch as of 427f357f0eed32c9ce17590ae9303a94e8b710e7 + the prs
https://github.com/ceph/ceph/pull/12001
https://github.com/ceph/ceph/pull/12167
https://github.com/ceph/ceph/pull/12067

CEPH_BRANCH=wip-jewel-10-2-4; MACHINE_NAME=smithi; SUITE_BRANCH=jewel; teuthology-suite --dry-run -v -c $CEPH_BRANCH -m $MACHINE_NAME --suite-branch jewel -k distro -s rados --subset 35/50 -e ceph-qa@ceph.com -p 100

Re-running failed jobs:

Rerunning failed jobs again

Further runs are still failing with similar errors, reported this issue as http://tracker.ceph.com/issues/18089

Rerunning failed tests ignoring valgrind exception as per Sage's comment 
CEPH_BRANCH=wip-jewel-10-2-4; MACHINE_NAME=smithi; SUITE_BRANCH=jewel; teuthology-suite -v -c $CEPH_BRANCH -m $MACHINE_NAME --suite-branch jewel -k distro -s upgrade/client-upgrade -e ceph-qa@ceph.com -p 100

Re ran after fixes were made to https://github.com/ceph/ceph-qa-suite/pull/1292 and merged

failures on ceph-create-keys
rerunning to avoid env noise as

Running on wip-ceph-coverage-jewel which is getting fixed as https://github.com/ceph/ceph-qa-suite/pull/1296

rgw

This is the jewel branch above + the pr https://github.com/ceph/ceph/pull/12207

teuthology-suite -k distro --priority 100 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel-avoid-xenial --email abhishek@suse.com --ceph wip-jewel-10-2-4-rc --machine-type smithi

Actions
Copy link
 #39

Updated by Sergey Jerusalimov over 7 years ago

Sergey Jerusalimov wrote:

Guys's please tell about https://www.spinics.net/lists/ceph-devel/msg33968.html

Actions
Copy link
 #40

Updated by Nathan Cutler over 7 years ago

@Sergey: That issue will be fixed in 10.2.5, see #18207

Actions
Copy link
 #41

Updated by Loïc Dachary over 7 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link

Also available in: Atom PDF