Ceph » Stable releases

Workflow¶

• Preparing the release
• Cutting the release
  • Loic gets approval from all leads
    • Yehuda, rgw: OK
    • Josh, RBD: OK
    • Sam, rados: OK
  • Loic informs Yuri that the branch is ready for testing
  • Yuri runs additional integration tests
  • If Yuri discovers new bugs with severity Critical, the relase goes back to being prepared, it was not ready after all
  • Sage writes and commits the release notes
  • Loic asks Sage if a point release should be published IN PROGRESS (note: if the decision is made to publish, all is ready for Alfredo to proceed)
  • Sage informs Alfredo that the branch is ready for release
  • Alfredo creates the packages and sets the release tag

teuthology runs on commit:3944c77c404c4a05886fe8276d5d0dd7e4f20410 (6 february)¶

Run snippet:

for suite in rbd rados rgw ; do
  ./virtualenv/bin/teuthology-suite --suite $suite --suite-branch dumpling --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --owner loic@dachary.org  --ceph dumpling-backports
done

pull/3552/head pull/3489/head pull/3439/head pull/3645/head pull/3655/head

rgw¶

• red http://pulpito.ceph.com/loic-2015-02-09_11:30:40-rgw-dumpling-backports---basic-multi/
  • http://pulpito.ceph.com/loic-2015-02-09_11:30:40-rgw-dumpling-backports---basic-multi/748360/ rgw/multifs/{clusters/fixed-2.yaml fs/ext4.yaml tasks/rgw_s3tests.yaml}
    • http://tracker.ceph.com/issues/10798 mkfs.ext4 fails with "apparently in use by the system"
    • http://tracker.ceph.com/issues/10797 ceph.py fallback fails because mkfs.ext4 does not have \-f option on trusty
  • http://pulpito.ceph.com/loic-2015-02-09_11:30:40-rgw-dumpling-backports---basic-multi/748356/ rgw/multifs/{clusters/fixed-2.yaml fs/btrfs.yaml tasks/rgw_s3tests.yaml}
    • mira114.stderr:Error: unable to open /dev/sdh: Device or resource busy
• green but red in the previous run http://pulpito.ceph.com/loic-2015-02-09_11:30:40-rgw-dumpling-backports---basic-multi/748370/ rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_s3tests.yaml validater/valgrind.yaml}
• green but red in the previous run http://pulpito.ceph.com/loic-2015-02-09_11:30:40-rgw-dumpling-backports---basic-multi/748369/ rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_swift.yaml validater/lockdep.yaml}

rbd¶

• red http://pulpito.ceph.com/loic-2015-02-06_16:26:38-rbd-dumpling-backports---basic-multi/
  • http://pulpito.ceph.com/loic-2015-02-06_16:26:38-rbd-dumpling-backports---basic-multi/742305/
    • xfs tests on btrfs ? http://tracker.ceph.com/issues/10795
    • maybe related to http://tracker.ceph.com/issues/4959

teuthology runs on commit:b5e871fe027db3b25f9129ae3e5b2326a89467c5 (6 february)¶

pull/3552/head pull/3489/head pull/3439/head pull/3645/head

rgw¶

• red http://pulpito.ceph.com/loic-2015-02-06_01:49:58-rgw-dumpling-backports---basic-multi/
  • apt-get update environmental failure
    • http://pulpito.ceph.com/loic-2015-02-06_01:49:58-rgw-dumpling-backports---basic-multi/741166 rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_s3tests.yaml validater/valgrind.yaml}
    • http://pulpito.ceph.com/loic-2015-02-06_01:49:58-rgw-dumpling-backports---basic-multi/741167 rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_swift.yaml validater/lockdep.yaml}

teuthology runs on commit:fd37e5a9749c5579a592ebe926a3c576be0617cd (3 february)¶

pull/3552/head pull/3489/head pull/3439/head

rgw¶

• green http://pulpito.ceph.com/loic-2015-02-03_14:46:09-rgw-dumpling-backports---basic-multi

rados¶

• green http://pulpito.ceph.com/loic-2015-02-03_14:41:24-rados-dumpling-backports---basic-multi

rbd¶

• red http://pulpito.ceph.com/loic-2015-02-03_14:40:23-rbd-dumpling-backports---basic-multi
  • http://pulpito.ceph.com/loic-2015-02-03_14:40:23-rbd-dumpling-backports---basic-multi/738732/
    • Can't reproduce http://tracker.ceph.com/issues/10773 qemu/tests/qemu-iotests/077 fails

teuthology runs on commit:e1c38bd5557ceb1411f9aa142deda28b459965e1 (25 january)¶

rados¶

• green on v0.67.11 http://pulpito.ceph.com/loic-2015-01-30_18:24:12-rados-v0.67.11---basic-multi/
  • http://pulpito.ceph.com/loic-2015-01-30_18:24:12-rados-v0.67.11---basic-multi/731949/ leveldb 1.12: hang on shutdown (happens rarely, can be ignored)
• red http://pulpito.ceph.com/loic-2015-01-29_15:41:06-rados-dumpling-backports---basic-multi
  • http://pulpito.ceph.com/loic-2015-01-29_15:41:06-rados-dumpling-backports---basic-multi/730305/
    • valgrind leak ReplicatedPG::get_object_context
• a lot of failures, environmental noise http://pulpito.ceph.com/loic-2015-01-27_17:08:20-rados-dumpling-backports---basic-multi/
  • [WRN] mon.2 10.214.137.124:6790/0 clock skew 1.21581s > max 0.5s
• killed because incorrectly selected teuthology dumpling branch http://pulpito.ceph.com/loic-2015-01-26_14:29:13-rados-dumpling-backports---basic-multi/

rgw¶

• green http://pulpito.ceph.com/loic-2015-01-29_15:47:17-rgw-dumpling-backports---basic-multi
• a lot of failures, environmental noise http://pulpito.ceph.com/loic-2015-01-27_17:13:16-rgw-dumpling-backports---basic-multi
  • [WRN] mon.2 10.214.137.124:6790/0 clock skew 1.21581s > max 0.5s
• killed because incorrectly selected teuthology dumpling branch http://pulpito.ceph.com/loic-2015-01-26_15:29:36-rgw-dumpling-backports---basic-multi/

rbd¶

• running http://pulpito.ceph.com/loic-2015-02-03_11:40:04-rbd-dumpling-backports---basic-multi/ with just rbd_fsx_cache_writethrough to reproduce it. The assumption is that it won't always happen.
• red http://pulpito.ceph.com/loic-2015-01-29_15:39:38-rbd-dumpling-backports---basic-multi
  • see mail thread workloads/rbd_fsx_cache_writethrough.yaml hangs on dumpling
  • it was there before, unrelated to the backports
• a lot of failures, environmental noise http://pulpito.ceph.com/loic-2015-01-27_17:07:21-rbd-dumpling-backports---basic-multi/
  • [WRN] mon.2 10.214.137.124:6790/0 clock skew 1.21581s > max 0.5s
  • wget -nv -O /home/ubuntu/cephtest/qemu/client.0.test.sh 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=qa/workunits/suites/tiobench.sh' && chmod 755 /home/ubuntu/cephtest/qemu/client.0.test.sh fixed by https://github.com/ceph/ceph-qa-suite/pull/317
• killed because incorrectly selected teuthology dumpling branch http://pulpito.ceph.com/loic-2015-01-26_15:51:29-rbd-dumpling-backports---basic-multi/

upgrade¶

• only env noise http://pulpito.front.sepia.ceph.com/ubuntu-2015-01-26_09:26:27-upgrade:dumpling-dumpling-distro-basic-vps/
• good run for reference http://pulpito.ceph.redhat.com/teuthology-2015-01-20_20:14:09-upgrade:dumpling-dumpling-distro-basic-magna/ (requires access to red hat VPN)

Inventory¶

http://workbench.dachary.org/ceph/ceph-backports/wikis/dumpling

QE Final validations¶

Running suites on the merged dumpling branch:

• http://pulpito.ceph.redhat.com/teuthology-2015-02-10_16:58:18-rados-dumpling-distro-basic-magna/
  • #10841 - infrastructure RESOLVED
  • #10840 - product
  • Job ['45215'] failed "Missing packages" - no ticket, Octo Lab connectivity issue
• http://pulpito.ceph.redhat.com/teuthology-2015-02-10_17:01:47-rbd-dumpling-distro-basic-magna/
  • #10842 - infrastructure
  • #10802 (related to #10680) - OK not to fix for the release per Josh
• http://pulpito.ceph.redhat.com/teuthology-2015-02-10_17:02:32-rgw-dumpling-distro-basic-magna/
  • #10843 (duplicate of #10801) - product
• http://pulpito.ceph.redhat.com/teuthology-2015-02-10_17:03:07-fs-dumpling-distro-basic-magna/
  • ALL PASSED
• http://pulpito.ceph.redhat.com/teuthology-2015-02-16_19:08:02-ceph-deploy-dumpling---basic-magna/
  • #10882 - - infrastructure RESOLVED
  • ALL PASSED
• http://pulpito.ceph.redhat.com/teuthology-2015-02-10_17:04:38-upgrade:dumpling-dumpling-distro-basic-magna/
  • #10694 - product "Won't fix" re-confirm
  • #10838 (duplicate of #10801) - product
• http://pulpito.ceph.redhat.com/teuthology-2015-02-18_17:24:07-upgrade:dumpling-dumpling-distro-basic-magna/ - still a problem, see #10838 (duplicate of #10801)
  • re-running http://pulpito.ceph.redhat.com/teuthology-2015-02-24_12:06:21-upgrade:dumpling-dumpling-distro-basic-magna/
  • PASSED final run http://pulpito.ceph.com/teuthology-2015-02-25_13:41:19-upgrade:dumpling-dumpling-distro-basic-multi/