Tasks #11492
Updated by Loïc Dachary almost 9 years ago
h3. Workflow
* "Preparing the release":http://ceph.com/docs/master/dev/development-workflow/#preparing-a-new-release *IN PROGRESS*
* "Cutting the release":http://ceph.com/docs/master/dev/development-workflow/#cutting-a-new-stable-release
** Abhishek asks Sage if a point release should be published *"YES":http://www.spinics.net/lists/ceph-devel/msg23944.html*
** Abhishek gets approval from all leads
*** Yehuda, rgw: (need to ask again, previous approval discussion "OK":http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/24847)
*** Gregory, CephFS: "OK":http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/24845
*** Josh, RBD: "OK":http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/24844
*** Sam, rados: "OK":http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/24846
** Sage writes and commits the release notes *IN PROGRESS*
** Abhishek informs Yuri that the branch is ready for testing
** Yuri runs additional integration tests
** If Yuri discovers new bugs with severity Critical, the release goes back to being prepared, it was not ready after all
** Yuri informs Alfredo that the branch is ready for release
** Alfredo creates the packages and sets the release tag
h3. Release information
** branch to build from: hammer, commit:???
** version: v0.94.2
** type of release: point release
** where to publish the release: debian/rpm-$release
h3. Teuthology run commit commit:823da44e24ce95c0afa9dd1be8d1df4f33396e2d (hammer-backports branch May 2015)
<code>git --no-pager log --format='%H %s' --graph ceph/hammer..ceph/hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'</code>
* + "Merge 4696: Wip 11622 hammer":https://github.com/ceph/ceph/commit/823da44e24ce95c0afa9dd1be8d1df4f33396e2d
* + "rgw: merge manifests correctly when there's prefix override":https://github.com/ceph/ceph/commit/293affe992118ed6e04f685030b2d83a794ca624
* + "rgw: restore buffer of multipart upload after EEXIST":https://github.com/ceph/ceph/commit/a43d24861089a02f3b42061e482e05016a0021f6
h4. rgw
Note: the *wip-11622-hammer* branch contains the same commits as *hammer-backports*
<pre>
filter='rgw/multifs/{overrides.yaml clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_readwrite.yaml}'
./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --filter="$filter" --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph wip-11622-hammer
</pre>
* *red* *running* http://pulpito.ceph.com/loic-2015-05-18_00:59:08-rgw-wip-11622-hammer---basic-multi/
** *known bug* (unrelated to rgw) "osd/ReplicatedPG.cc: 2141: FAILED assert(op == prdop->op)":http://tracker.ceph.com/issues/11211
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --filter-out btrfs,ext4,apache --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph wip-11622-hammer</code>
* *red* http://pulpito.ceph.com/loic-2015-05-16_01:48:25-rgw-wip-11622-hammer---basic-multi/
** *environmental noise* "rgw/multifs/{overrides.yaml clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_readwrite.yaml}":http://pulpito.ceph.com/loic-2015-05-16_01:48:25-rgw-wip-11622-hammer---basic-multi/894283/ (the machine apparently died, the log end abruptly)
h3. Teuthology run commit commit:a79146fc3cae28bf4c07478fb4566b06942da60d (hammer-backports branch May 2015)
h4. rados
Note: rados suite must run on "branch wip-11429-hammer":https://github.com/ceph/ceph-qa-suite/pull/432 because it is required by "OSD::load_pgs: we need to handle the case where an upgrade from earlier versions which ignored non-existent pgs resurrects a pg with a prehistoric osdmap":http://tracker.ceph.com/issues/11429
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite rados --subset 0/7 --filter-out btrfs,ext4 --suite-branch wip-11429-hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports</code>
* *green* http://pulpito.ceph.com/loic-2015-05-07_09:34:21-rados-hammer-backports---basic-multi/
h4. fs
Note: the following run contains https://github.com/ceph/ceph/pull/4629 in addition to the commits above. It was scheduled after they were merged, except for one pull request related to rgw and with no consequence on fs testing (https://github.com/ceph/ceph/pull/4568).
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite fs -k testing --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email greg@gregs42.com --ceph hammer-uclient-checking</code>
* *green* http://pulpito.ceph.com/loic-2015-05-09_11:11:23-fs-hammer-uclient-checking-testing-basic-multi/
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite fs -k testing --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports</code>
* *red* http://pulpito.ceph.com/loic-2015-05-07_09:44:39-fs-hammer-backports-testing-basic-multi/
** *environmental noise* sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes ... or hung
*** "fs/basic/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/mds_creation_retry.yaml}":http://pulpito.ceph.com/loic-2015-05-07_09:44:39-fs-hammer-backports-testing-basic-multi/878752
*** " fs/basic/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/cfuse_workunit_suites_ffsb.yaml}":http://pulpito.ceph.com/loic-2015-05-07_09:44:39-fs-hammer-backports-testing-basic-multi/878721/
h4. rgw
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --filter-out btrfs,ext4,apache --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports</code>
* *green* http://pulpito.ceph.com/loic-2015-05-07_09:21:24-rgw-hammer-backports---basic-multi/
h4. rbd
<code>./virtualenv/bin/teuthology-suite --priority 101 --suite rbd --filter-out btrfs,ext4 --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --ceph hammer-backports</code>
* *red* http://pulpito.ceph.com/loic-2015-05-07_09:46:27-rbd-hammer-backports---basic-multi
** *can be ignored* "librbd: crash when two clients try to write to an exclusive locked image":http://tracker.ceph.com/issues/11537 rare enough, pre-existing)
*** "rbd/thrash/{base/install.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/rbd_fsx_cache_writeback.yaml}":http://pulpito.ceph.com/loic-2015-05-07_09:46:27-rbd-hammer-backports---basic-multi/878799
h3. Teuthology run commit d7828f7a8dcff7ec5a1a8458e6a610c0a07867ca (hammer-backports branch May 2015)
<code>git --no-pager log --format='%H %s' --graph ceph/hammer..ceph/hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'</code>
* + "Merge 4515: rgw: critical fixes for hammer":https://github.com/ceph/ceph/commit/d7828f7a8dcff7ec5a1a8458e6a610c0a07867ca
* + "Merge 4496: librbd: deadlock in image refresh":https://github.com/ceph/ceph/commit/2ae10fe87dc40721705efe2050711cac157fb18b
* |\
* | + "librbd: acquire cache_lock before refreshing parent":https://github.com/ceph/ceph/commit/0cdc93fbdcf68a31e6aada38b0cb9d66efdc512d
* | + "librados_test_stub: AIO operation callbacks should be via Finisher":https://github.com/ceph/ceph/commit/a1b4aeb8e8c3a8d5c8284dcee8e03f501a77928c
* + "Merge 4497: Objectcacher setting max object counts too low":https://github.com/ceph/ceph/commit/a4c50f8d428c8e86479b6ebd21c9e8617561e32d
* |\
* | + "librbd: updated cache max objects calculation":https://github.com/ceph/ceph/commit/2864da86ca689472341e8a80f7ad6fcc5eb8321a
* + "Merge 4498: librbd notification race condition on snap_create":https://github.com/ceph/ceph/commit/080135810207759bb8008e97fb15f6c924fd9ba6
* |\
* | + "librbd: moved snap_create header update notification to initiator":https://github.com/ceph/ceph/commit/0e6a032c9e6ddae20be82df7500a0758d2cd8e74
* + "Merge 4499: ImageWatcher should cancel in-flight ops on watch error":https://github.com/ceph/ceph/commit/d93e440c8d9cb27d50c54d2e3d83999690c8f532
* |\
* | + "librbd: ImageWatcher should cancel in-flight ops on watch error":https://github.com/ceph/ceph/commit/c6159724f065731c41b2d29a48d0f0a3dc82340b
* + "Merge 4500: rgw : make quota/gc thread configurable for starting":https://github.com/ceph/ceph/commit/df7d55aa5649fe8c57d7be04990e2001f8771475
* |\
* | + "rgw - make starting quota/gc threads configurable":https://github.com/ceph/ceph/commit/c6edc16fee027f6cdefacba08f1edc436a7406c5
* + "Merge 4501: Improve rgw HEAD request by avoiding read the body of the first chunk":https://github.com/ceph/ceph/commit/5c71f68f15e250aae8f88011551d8da626ab7baf
* |\
* | + "rgw: do not pre-fetch data for HEAD requests":https://github.com/ceph/ceph/commit/2f34d2e73bb4bded4779af15a337c75eb2d1497f
* + "Merge 4502: objecter: can get stuck in redirect loop if osdmap epoch == last_force_op_resend":https://github.com/ceph/ceph/commit/293ecdfa47cbbd37f107fbddfd39c766111edb75
* |\
* | + "osdc: add epoch_t last_force_resend in Op/LingerOp.":https://github.com/ceph/ceph/commit/8d9f4d4eb546e26eeb3911811bdeb166d06cb1d1
* + "Merge 4503: use a new disk as journal disk,ceph-disk prepare fail":https://github.com/ceph/ceph/commit/6cecb3f64a35786848eb395c0d367dda33862339
* |\
* | + "ceph-disk: more robust parted output parser":https://github.com/ceph/ceph/commit/8709e34f931809f7129cdac1203ec4e774e3eb4c
* + "Merge 4504: ceph-dencoder links to libtcmalloc, and shouldn't":https://github.com/ceph/ceph/commit/5e0ef766320b1a9e4baa2c59d11b9f197a2da88c
* |\
* | + "Move ceph-dencoder build to client":https://github.com/ceph/ceph/commit/478968670caecd67f7995a09b60f6208729e3de3
* | + "Rework mds/Makefile.am to support a dencoder client build":https://github.com/ceph/ceph/commit/7eabb70b906e50901551ab96453f05fe662a8876
* | + "rgw/Makefile.am: Populate DENCODER_SOURCES properly":https://github.com/ceph/ceph/commit/b903ad28a68772fa0b7a88b4db2724f4d07565d5
* | + "Dencoder should never be built with tcmalloc":https://github.com/ceph/ceph/commit/f994483c0e3a60226c8fb6983380ef8400b0160e
* + "Merge 4505: rgw - improve performance for large object (multiple chunks) GET":https://github.com/ceph/ceph/commit/12b944a7a6635115838e71bb102eebd230a292d1
* |\
* | + "rgw : Issue AIO for next chunk first before flush the (cached) data.":https://github.com/ceph/ceph/commit/379ef714f7149a748891dafd41db80c247d35975
* + "Merge 4507: compilation error: No high-precision counter available (armhf, powerpc..)":https://github.com/ceph/ceph/commit/c1ae8feed1914585dc2b94867fc181a854ec682a
* + "Add support for PPC architecture, provide fallback":https://github.com/ceph/ceph/commit/da7f6835b15370ce0120a64f7ac3359f3ba4729b
h4. rgw
<pre>
./virtualenv/bin/teuthology-suite --priority 101 --suite rgw --filter-out btrfs,ext4,apache --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph hammer-backports
</pre>
* *green* http://pulpito.ceph.com/loic-2015-05-03_14:50:25-rgw-hammer-backports---basic-multi/
h4. fs
<pre>
run=loic-2015-05-04_11:27:13-fs-hammer-backports---basic-multi
filter='fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-flush.yaml}'
./virtualenv/bin/teuthology-suite --filter="$filter" -k testing --priority 101 --suite fs --suite-branch wip-10712-hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports
</pre>
* *green* http://pulpito.ceph.com/loic-2015-05-05_11:54:53-fs-hammer-backports-testing-basic-multi/
<pre>
run=loic-2015-05-03_14:58:27-fs-hammer-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 101 --suite fs --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-05-04_11:27:13-fs-hammer-backports---basic-multi/
** "TestFlush intermittent failure on scatter_writebehind event":http://tracker.ceph.com/issues/10712 Test failure: test_flush (tasks.mds_flush.TestFlush)
*** "fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-flush.yaml}":http://pulpito.ceph.com/loic-2015-05-04_11:27:13-fs-hammer-backports---basic-multi/874954
<pre>
./virtualenv/bin/teuthology-suite --priority 101 --suite fs --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-05-03_14:58:27-fs-hammer-backports---basic-multi/
** *Test failure: test_flush (tasks.mds_flush.TestFlush)*
*** "fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-flush.yaml}":http://pulpito.ceph.com/loic-2015-05-03_14:58:27-fs-hammer-backports---basic-multi/873887
** *Test failure: test_client_pin (tasks.mds_client_limits.TestClientLimits)*
*** "fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/client-limits.yaml}":http://pulpito.ceph.com/loic-2015-05-03_14:58:27-fs-hammer-backports---basic-multi/873885
h4. rbd
<pre>
./virtualenv/bin/teuthology-suite --priority 101 --suite rbd --filter-out btrfs,ext4 --suite-branch hammer --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports
</pre>
* *green* http://pulpito.ceph.com/loic-2015-05-03_14:55:24-rbd-hammer-backports---basic-multi/
h4. rados
<pre>
$ git ls-remote https://github.com/dachary/ceph-qa-suite.git wip-rados-no-thrash-hammer
8233eae8329d1ef80dece59a2fe0518c13e51135 refs/heads/wip-rados-no-thrash-hammer
$ git clone -b wip-rados-no-thrash-hammer https://github.com/dachary/ceph-qa-suite.git /tmp/ceph-qa-suite
$ ./virtualenv/bin/teuthology-suite --filter-out ext4,btrfs --priority 101 --suite rados --suite-dir /tmp/ceph-qa-suite --machine-type plana,burnupi,mira --distro ubuntu --email abhishek.lekshmanan@gmail.com --owner loic@dachary.org --ceph hammer-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-05-04_11:18:05-rados-hammer-backports---basic-multi/
** *manager*
*** "rados/thrash-erasure-code/{clusters/fixed-2.yaml fs/xfs.yaml workloads/ec-small-objects.yaml}":http://pulpito.ceph.com/loic-2015-05-04_11:18:05-rados-hammer-backports---basic-multi/874949
*** "rados/thrash-erasure-code/{clusters/fixed-2.yaml fs/xfs.yaml workloads/ec-radosbench.yaml}":http://pulpito.ceph.com/loic-2015-05-04_11:18:05-rados-hammer-backports---basic-multi/874948
*** "rados/thrash-erasure-code/{clusters/fixed-2.yaml fs/xfs.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml}":http://pulpito.ceph.com/loic-2015-05-04_11:18:05-rados-hammer-backports---basic-multi/874947
*** "rados/thrash-erasure-code/{clusters/fixed-2.yaml fs/xfs.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}":http://pulpito.ceph.com/loic-2015-05-04_11:18:05-rados-hammer-backports---basic-multi/874946
h3. QE Validation
| *Suite* | *Runs/Reruns* | *Notes/Issues* |
| rados | http://pulpito.ceph.com/teuthology-2015-05-11_17:21:36-rados-hammer-distro-basic-multi | |
| | http://pulpito.ceph.com/teuthology-2015-05-13_07:14:55-rados-hammer-distro-basic-multi/ | |
| | http://pulpito.ceph.redhat.com/teuthology-2015-05-11_20:22:13-rados-hammer-distro-basic-magna/ | |
| | http://pulpito.ceph.redhat.com/teuthology-2015-05-15_16:15:16-rados-hammer-distro-basic-magna/ | |
| | | |
| | typica 2/3 http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-11_21:02:36-rados-hammer-distro-basic-typica/ | |
| | http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-15_12:42:53-rados-hammer-distro-basic-typica/ | |
|rbd | http://pulpito.ceph.com/teuthology-2015-05-12_17:53:19-rbd-hammer-distro-basic-multi/ | PASSED |
| | http://pulpito.ceph.com/teuthology-2015-05-13_14:59:24-rbd-hammer-distro-basic-multi/ | rerun failed |
|rgw | http://pulpito.ceph.com/teuthology-2015-05-12_17:42:50-rgw-hammer-distro-basic-multi/ | blocked by #11591, #11604, #11622, #11613 |
| | http://pulpito.ceph.com/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/ | rerun failed |
|fs | http://pulpito.ceph.com/teuthology-2015-05-13_15:03:35-fs-hammer-testing-basic-multi/ | |
| | http://pulpito.ceph.com/teuthology-2015-05-14_14:11:49-fs-hammer-testing-basic-multi/ | |
| | http://pulpito.ceph.com/teuthology-2015-05-15_09:04:54-fs-hammer-testing-basic-multi/ | |
| | http://pulpito.ceph.com/teuthology-2015-05-15_13:23:06-fs-hammer-testing-basic-multi/ | |
| | | |
|krbd | http://pulpito.ceph.com/teuthology-2015-05-13_15:07:10-krbd-hammer-testing-basic-multi/ | |
|kcephfs | http://pulpito.ceph.com/teuthology-2015-05-13_15:08:50-kcephfs-hammer-testing-basic-multi/ | |
|knfs | http://pulpito.ceph.com/teuthology-2015-05-13_15:11:10-knfs-hammer-testing-basic-multi/ | |
|hadoop | http://pulpito.ceph.redhat.com/teuthology-2015-05-13_18:12:35-hadoop-hammer-testing-basic-magna/ | |
|rest | http://pulpito.ceph.redhat.com/teuthology-2015-05-13_18:16:34-rest-hammer-testing-basic-magna/ | |
|multimds | | optional |
|samba | http://pulpito.ceph.com/teuthology-2015-05-13_15:20:03-samba-hammer-testing-basic-multi/ | |
|ceph-deploy | http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-13_18:31:05-ceph-deploy-hammer-distro-basic-typica/ | |
|upgrade/client-upgrade | http://pulpito.ceph.redhat.com/teuthology-2015-05-13_18:39:57-upgrade:client-upgrade-hammer-distro-basic-magna/ | |
|upgrade/dumpling-firefly-x | http://pulpito.ceph.com/teuthology-2015-05-13_15:44:01-upgrade:dumpling-firefly-x-hammer-distro-basic-multi/ | |
|upgrade/firefly-x | http://pulpito.ceph.redhat.com/teuthology-2015-05-13_18:45:13-upgrade:firefly-x-hammer-distro-basic-magna/ | |
|upgrade/giant-x | http://pulpito.ceph.com/teuthology-2015-05-11_17:05:01-upgrade:giant-x-hammer-distro-basic-vps/ | PASSED |
| | http://pulpito.ceph.com/teuthology-2015-05-12_13:39:24-upgrade:giant-x-hammer-distro-basic-vps/ | rerun failed |
|upgrade/hammer | http://pulpito.ceph.redhat.com/teuthology-2015-05-08_13:10:30-upgrade:hammer-hammer-distro-basic-magna/ | #11570 |
|powercycle | http://pulpito.ceph.com/teuthology-2015-05-11_16:26:54-powercycle-hammer-testing-basic-multi/ | #11204, https://github.com/ceph/ceph-qa-suite/pull/439 |
| | http://pulpito.ceph.com/teuthology-2015-05-13_16:10:46-powercycle-hammer-testing-basic-multi/ | rerun failed |