Tasks #11153
Updated by Loïc Dachary about 9 years ago
h3. Workflow * "Preparing the release":http://ceph.com/docs/master/dev/development-workflow/#preparing-a-new-release *IN PROGRESS* * "Cutting the release":http://ceph.com/docs/master/dev/development-workflow/#cutting-a-new-stable-release ** Loic gets approval from all leads *** Yehuda, rgw: *** Gregory, CephFS: *** Josh, RBD: *** Sam, rados: ** Sage writes and commits the release notes ** Loic informs Yuri that the branch is ready for testing ** Yuri runs additional integration tests ** If Yuri discovers new bugs with severity Critical, the relase goes back to being prepared, it was not ready after all ** Yuri informs Alfredo that the branch is ready for release ** Alfredo creates the packages and sets the release tag h3. Release information * branch to build from: giant, commit:??? * version: v0.87.2 * type of release: point release * where to publish the release: debian/rpm-$release h3. Synchronized pull requests * "rgw acl response should start with <?xml version...":http://tracker.ceph.com/issues/10106 https://github.com/ceph/ceph-qa-suite/pull/370 https://github.com/ceph/ceph/pull/4082 h3. Inventory * http://workbench.dachary.org/ceph/ceph-backports/wikis/giant * waiting : * OK : * Not OK : h3. teuthology run commit:b6bebbbc99540b76221aeccb3693784b414f607b (integration branch, march (march 2015) The is one potential regression (does not happen all the time but appears to be new) : "osdc/Objecter.cc: 405: FAILED assert(tick_event == __null)":http://tracker.ceph.com/issues/11183 h4. rados <pre> run=loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *green* http://pulpito.ceph.com/loic-2015-03-20_17:38:25-rados-giant-backports---basic-multi/ <code>./virtualenv/bin/teuthology-suite --filter-out btrfs,ext4 --priority 1000 --suite rados --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code> * *red* http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi ** *environmental noise* NTP problems : cluster [WRN] message from mon.1 was stamped 0.897801s in the future, clocks not synchronized" in cluster log *** "rados/thrash-erasure-code-isa/{clusters/fixed-2.yaml distros/rhel_7.0.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}":http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi/811652 ** *new bug* "osdc/Objecter.cc: 405: FAILED assert(tick_event == __null)":http://tracker.ceph.com/issues/11183 *** "rados/monthrash/{ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml}":http://pulpito.ceph.com/loic-2015-03-19_19:01:57-rados-giant-backports---basic-multi/811322 h4. rgw <pre> run=loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-regional-summary --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *green* *running* http://pulpito.ceph.com/loic-2015-03-22_20:48:29-rgw-giant-backports---basic-multi/ <pre> run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-regional-summary --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *red* http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/ ** *environmental error* Download error on https://pypi.python.org/simple/ *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815909 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815907 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815905 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815903 ** *needs fixing* Remote branch wip-rgw-regional-summary not found in upstream origin, using HEAD instead, because the wip-rgw-regional-summary branch does not exist *** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815901 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815899 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815897 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815895 ** *needs fixing* 'git clone -b wip-rgw-regional-summary git://ceph.com/git/s3-tests.git /home/ubuntu/cephtest/s3-tests' because the wip-rgw-regional-summary branch does not exist *** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815902 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/civetweb.yaml fs/xfs.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815900 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815898 *** "rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_s3tests.yaml}":http://pulpito.ceph.com/loic-2015-03-22_19:18:01-rgw-giant-backports---basic-multi/815896 Before v0.87.1 was released 6b08a729540c61f3c8b15c5a3ce9382634bf800c was tested at http://tracker.ceph.com/issues/10501#rgw and came back ok. There was a few commits merged in between: <pre> git log --no-merges --oneline tags/v0.87.1 ^6b08a729540c61f3c8b15c5a3ce9382634bf800c^ 283c2e7 0.87.1 734e9af osd: tolerate sessionless con in fast dispatch path ccb0914 qa: use correct binary path on rpm-based systems df8285c fsync-tester: print info about PATH and locations of lsof lookup 91515e7 osdc: Constrain max number of in-flight read requests </pre> Later commit:ccb0914f76da23acdd7374233cd1939ab80ef3c8 was tested at http://pulpito.ceph.com/teuthology-2015-02-12_17:41:04-rgw-giant-distro-basic-multi/ and was green. Two commits were merged after that and before v0.87.1 was released: <pre> $ git log --oneline tags/v0.87.1 ^ccb0914f76da23acdd7374233cd1939ab80ef3c8^ 283c2e7 0.87.1 4178e32 Merge pull request #3731 from liewegas/wip-10834-giant 734e9af osd: tolerate sessionless con in fast dispatch path ccb0914 qa: use correct binary path on rpm-based systems </pre> <pre> filter='rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}' ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph v0.87.1 </pre> The problem seems to be related to "rgwadmin tasks assumes non-regional output":http://tracker.ceph.com/issues/11180 that was fixed two days ago. But why did it manifest itself all of a sudden and changes things for a branch that was otherwise unmodified ? * *red* http://pulpito.ceph.com/loic-2015-03-22_14:55:18-rgw-v0.87.1---basic-multi/ (v0.87.1) <pre> filter='rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}' ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant </pre> * *red* http://pulpito.ceph.com/loic-2015-03-22_12:55:06-rgw-giant---basic-multi/ (giant branch) <pre> run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *red* http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/ ** *???* tasks/radosgw_admin.py assert time.time() - timestamp <= (20 x 60) *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815807 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815806 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815805 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815803 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815802 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815801 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815798 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815797 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815796 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815795 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815794 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815792 ** *environmental noise* Command failed with status 1: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap' *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815800 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815793 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815799 ** *environmental noise* No route to host (113) plana59 is down *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-22_10:53:10-rgw-giant-backports---basic-multi/815804 <pre> run=loic-2015-03-20_00:17:55-rgw-giant-backports---basic-multi eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/jobs/?status=fail | jq '.[].description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//') ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *red* http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/ ** the s3-tests referred to a branch wip-acl-giant which didn't exist in https://github.com/ceph/s3-tests, "pushed the wip-acl-giant branch":https://github.com/ceph/s3-tests/tree/wip-rgw-acl-giant and trying again ** *tasks/radosgw_admin.py assert time.time() - timestamp <= (20 x 60) * *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813491 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813490 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813489 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813488 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813487 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813486 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813485 *** "rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813484 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813483 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813482 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813481 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813480 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/replicated.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813479 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813478 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-profile.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813477 *** "rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml rgw_pool_type/ec-cache.yaml}":http://pulpito.ceph.com/loic-2015-03-20_17:58:01-rgw-giant-backports---basic-multi/813476 <code>./virtualenv/bin/teuthology-suite --priority 1000 --suite rgw --filter-out btrfs,ext4 --suite-branch wip-rgw-acl-giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code> * *red* http://pulpito.ceph.com/loic-2015-03-19_18:55:59-rgw-giant-backports---basic-multi ** assuming that all 24 failed jobs are environmental noise h4. rbd <pre> filter='rbd/librbd/{cache/none.yaml cachepool/small.yaml clusters/fixed-3.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}' ./virtualenv/bin/teuthology-suite --filter="$filter" --priority 1000 --suite rbd --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports </pre> * *green* http://pulpito.ceph.com/loic-2015-03-20_17:46:39-rbd-giant-backports---basic-multi/ <code>./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code> * *red* http://pulpito.ceph.com/loic-2015-03-19_18:53:40-rbd-giant-backports---basic-multi ** *environmental noise* plana34.stdout:[15188.076023] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:12689] *** "rbd/librbd/{cache/none.yaml cachepool/small.yaml clusters/fixed-3.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}":http://pulpito.ceph.com/loic-2015-03-19_18:53:40-rbd-giant-backports---basic-multi/811021/ h4. fs <code>./virtualenv/bin/teuthology-suite -k testing --priority 1000 --suite fs --suite-branch giant --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org --ceph giant-backports</code> * *green* http://pulpito.ceph.com/loic-2015-03-19_18:52:05-fs-giant-backports-testing-basic-multi