Tasks #11153
Updated by Loïc Dachary about 9 years ago
h3. Workflow
* "Preparing the release":http://ceph.com/docs/master/dev/development-workflow/#preparing-a-new-release *IN PROGRESS*
* "Cutting the release":http://ceph.com/docs/master/dev/development-workflow/#cutting-a-new-stable-release
** Loic gets approval from all leads
*** Yehuda, rgw:
*** Gregory, CephFS:
*** Josh, RBD:
*** Sam, rados:
** Sage writes and commits the release notes
** Loic informs Yuri that the branch is ready for testing
** Yuri runs additional integration tests
** If Yuri discovers new bugs with severity Critical, the relase goes back to being prepared, it was not ready after all
** Yuri informs Alfredo that the branch is ready for release
** Alfredo creates the packages and sets the release tag
h3. Release information
* branch to build from: giant, commit:???
* version: v0.87.2
* type of release: point release
* where to publish the release: debian/rpm-$release
h3. Synchronized pull requests
* "rgw acl response should start with <?xml version...":http://tracker.ceph.com/issues/10106 https://github.com/ceph/ceph-qa-suite/pull/370 https://github.com/ceph/ceph/pull/4082
h3. Inventory
* http://workbench.dachary.org/ceph/ceph-backports/wikis/giant
* waiting :
* OK :
* Not OK :
h3. teuthology run commit:b6bebbbc99540b76221aeccb3693784b414f607b (march 2015)
h4. rados
<pre>
run=loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *green* http://pulpito.ceph.com/loic-2015-03-22_14:55:18-rgw-v0.87.1---basic-multi/
<code>./virtualenv/bin/teuthology-suite --filter-out btrfs,ext4 --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code>
* *red* http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi
** *environmental noise* NTP problems : cluster [WRN] message from mon.1 was stamped 0.897801s in the future, clocks not synchronized" in cluster log
*** "rados/thrash-erasure-code-isa/{clusters/fixed-2.yaml distros/rhel_7.0.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}":http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi/811652
** *new bug* "osdc/Objecter.cc: 405: FAILED assert(tick_event == __null)":http://tracker.ceph.com/issues/11183
*** "rados/monthrash/{ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml}":http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi/811322
h4. rgw
<pre>
run=loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-regional-summary --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *running* http://pulpito.ceph.com/loic-2015-03-22_20:48:29-rgw-giant-backports---basic-multi/
<pre>
run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-regional-summary --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/
** *environmental error* Download error on https://pypi.python.org/simple/
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815909
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815907
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815905
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815903
** *needs fixing* Remote branch wip-rgw-regional-summary not found in upstream origin, using HEAD instead, because the wip-rgw-regional-summary branch does not exist *"S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'"*
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815901
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815899
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815897
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815895
** *needs fixing* 'git clone -b wip-rgw-regional-summary git://ceph.com/git/s3-tests.git /home/ubuntu/cephtest/s3-tests' because the wip-rgw-regional-summary branch does not exist
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815902
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815900
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815898
*** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815896
Before v0.87.1 was released 6b08a729540c61f3c8b15c5a3ce9382634bf800c was tested at http://tracker.ceph.com/issues/10501#rgw and came back ok. There was a few commits merged in between:
<pre>
git log --no-merges --oneline tags/v0.87.1 ^6b08a729540c61f3c8b15c5a3ce9382634bf800c^
283c2e7 0.87.1
734e9af osd: tolerate sessionless con in fast dispatch path
ccb0914 qa: use correct binary path on rpm-based systems
df8285c fsync-tester: print info about PATH and locations of lsof lookup
91515e7 osdc: Constrain max number of in-flight read requests
</pre>
Later commit:ccb0914f76da23acdd7374233cd1939ab80ef3c8 was tested at http://pulpito.ceph.com/teuthology-2015-02-12_17:41:04-rgw-giant-distro-basic-multi/ and was green. Two commits were merged after that and before v0.87.1 was released:
<pre>
$ git log --oneline tags/v0.87.1 ^ccb0914f76da23acdd7374233cd1939ab80ef3c8^
283c2e7 0.87.1
4178e32 Merge pull request #3731 from liewegas/wip-10834-giant
734e9af osd: tolerate sessionless con in fast dispatch path
ccb0914 qa: use correct binary path on rpm-based systems
</pre>
<pre>
filter='rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}'
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph v0.87.1
</pre>
The problem seems to be related to "rgwadmin tasks assumes non-regional output":http://tracker.ceph.com/issues/11180 that was fixed two days ago. But why did it manifest itself all of a sudden and changes things for a branch that was otherwise unmodified ?
* *red* http://pulpito.ceph.com/loic-2015-03-22_14:55:18-rgw-v0.87.1---basic-multi/ (v0.87.1)
<pre>
filter='rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}'
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant
</pre>
* *red* http://pulpito.ceph.com/loic-2015-03-22_12:55:06-rgw-giant---basic-multi/ (giant branch)
<pre>
run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/
** *???* tasks/radosgw_admin.py assert time.time() - timestamp <= (20 x 60)
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815807
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815806
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815805
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815803
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815802
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815801
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815798
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815797
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815796
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815795
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815794
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815792
** *environmental noise* Command failed with status 1: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815800
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815793
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815799
** *environmental noise* No route to host (113) plana59 is down
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815804
<pre>
run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *red* http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/
** the s3-tests referred to a branch wip-acl-giant which didn't exist in https://github.com/ceph/s3-tests, "pushed the wip-acl-giant branch":https://github.com/ceph/s3-tests/tree/wip-rgw-acl-giant and trying again
** *tasks/radosgw_admin.py assert time.time() - timestamp <= (20 x 60) *
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813491
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813490
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813489
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813488
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813487
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813486
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813485
*** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813484
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813483
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813482
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813481
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813480
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813479
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813478
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813477
*** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813476
<code>./virtualenv/bin/teuthology-suite --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code>
* *red* http://pulpito.ceph.com/loic-2015-03-19_18:55:59-rgw-giant-backports---basic-multi
** assuming that all 24 failed jobs are environmental noise
h4. rbd
<pre>
filter='rbd/librbd/{cache/none.yaml cachepool/small.yaml clusters/fixed-3.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}'
./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rbd --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports
</pre>
* *green* http://pulpito.ceph.com/loic-2015-03-20_17:46:39-rbd-giant-backports---basic-multi/
<code>./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code>
* *red* http://pulpito.ceph.com/loic-2015-03-19_18:53:40-rbd-giant-backports---basic-multi
** *environmental noise* plana34.stdout:[15188.076023] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:12689]
*** "rbd/librbd/{cache/none.yaml cachepool/small.yaml clusters/fixed-3.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}":http://pulpito.ceph.com/loic-2015-03-19_18:53:40-rbd-giant-backports---basic-multi/811021/
h4. fs
<code>./virtualenv/bin/teuthology-suite -k testing --priority 1000 --suite fs --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code>
* *green* http://pulpito.ceph.com/loic-2015-03-19_18:52:05-fs-giant-backports-testing-basic-multi