Tasks #17487
jewel v10.2.4
0%
Description
Workflow¶
- Preparing the release
- Cutting the release
- Abhishek V. asks Abhishek L. if a point release should be published YES
- Abhishek V. gets approval from all leads
- Abhishek L. writes and commits the release notes, including:
Set require_jewel_osds osdmap flag on upgrade from hammer When the last hammer OSD in a cluster containing jewel MONs is upgraded to jewel, as of 10.2.4 the jewel MONs will issue this warning: "all OSDs are running jewel or later but the 'require_jewel_osds' osdmap flag is not set" and change the cluster health status to HEALTH_WARN. This is a signal for the admin to do "ceph osd set require_jewel_osds" - by doing this, the admin acknowledges that there is no downgrade path.
- Abhishek V. informs Yuri that the branch is ready for testing DONE
- Yuri runs additional integration tests IN PROGRESS
- If Yuri discovers new bugs that need to be backported urgently (i.e. their priority is set to Urgent), the release goes back to being prepared, it was not ready after all
- Yuri informs Alfredo that the branch is ready for release
- Alfredo creates the packages and sets the release tag
Release information¶
- branch to build from: jewel, commit: 427f357f0eed32c9ce17590ae9303a94e8b710e7
- version: v10.2.4
- type of release: point release
- where to publish the release: http://download.ceph.com/debian-jewel and http://download.ceph.com/rpm-jewel
History
#1 Updated by Loïc Dachary over 7 years ago
git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
- + Pull request 9388
- |\
- | + mon : Display full flag in ceph status if full flag is set
- + Pull request 9405
- + Pull request 9739
- |\
- | + msg/msg_types: update sockaddr, sockaddr_storage accessors
- | + osd: add peer_addr in heartbeat_check log message
- + Pull request 10107
- |\
- | + client: kill QuotaTree
- + Pull request 10278
- |\
- | + common: fix value of CINIT_FLAG_DEFER_DROP_PRIVILEGES
- + Pull request 10496
- |\
- | + crush: reset bucket->h.items[i] when removing tree item
- + Pull request 10645
- |\
- | + librbd: journal::Replay no longer holds lock while completing callback
- + Pull request 10650
- |\
- | + librbd: prevent creation of clone from non-primary mirrored image
- + Pull request 10652
- |\
- | + librbd: prevent creation of v2 image ids that are too large
- + Pull request 10654
- + Pull request 10656
- + Pull request 10657
- + Pull request 10658
- + Pull request 10659
- + Pull request 10660
- + Pull request 10662
- + Pull request 10663
- + Pull request 10664
- + Pull request 10757
- |\
- | + mon/osdmonitor: decouple adjust_heartbeat_grace and min_down_reporters
- + Pull request 10758
- |\
- | + ceph-disk: timeout ceph-disk to avoid blocking forever
- + Pull request 10759
- |\
- | + mon/osdmonitor: fix incorrect output of osd df due to osd out
- | + mon/osdmonitor: initialize local variable kb_avail_i
- + Pull request 10760
- + Pull request 10761
- |\
- | + fast dispatch heartbeat message
- + Pull request 10763
- + Pull request 10764
- + Pull request 10765
- + Pull request 10784
- |\
- | + librbd: request exclusive lock if current owner cannot execute op
- + Pull request 10793
- |\
- | + librbd: support deleting image by id instead of name
- | + rbd-mirror: use pool id + global image id as deletion primary key
- + Pull request 10794
- |\
- | + LTTng-UST disabled for openSUSE
- + Pull request 10796
- |\
- | + rbd: bench io-size should not be larger than image size
- + Pull request 10804
- + Pull request 10812
- |\
- | + ceph-osd-prestart.sh: check existence of OSD data directory
- + Pull request 10832
- |\
- | + rgw: Add documentation for RGW default quota
- | + rgw: Do not bother to check against conf values during quota handling
- | + rgw: Let the default quota settings take effect during user creation
- + Pull request 10845
- + Pull request 10847
#2 Updated by Loïc Dachary over 7 years ago
rbd¶
teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
INFO:teuthology.suite:Passed subset=2/5
- fail http://pulpito.ceph.com/loic-2016-08-25_06:42:29-rbd-jewel-backports---basic-smithi
- 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0'
- '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T12:02:03.995823+00:00 smithi023 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T12:50:49.813976+00:00 smithi005 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_FEATURES=125 VALGRIND=helgrind adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_python.sh' - '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T11:37:36.532331+00:00 smithi024 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T12:30:58.749523+00:00 smithi023 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_FEATURES=125 VALGRIND=helgrind adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd.sh' - '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T12:21:12.832668+00:00 smithi003 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - 'mkdir
p -/home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_ARGS=\'\' RBD_MIRROR_USE_RBD_MIRROR=1 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.cluster1.client.mirror/rbd/rbd_mirror_stress.sh'- rbd/mirror/{base/install.yaml cluster/{2-node.yaml openstack.yaml} fs/xfs.yaml msgr-failures/many.yaml rbd-mirror/one-per-cluster.yaml workloads/rbd-mirror-stress-workunit.yaml}
- rbd/mirror/{base/install.yaml cluster/{2-node.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml rbd-mirror/one-per-cluster.yaml workloads/rbd-mirror-stress-workunit.yaml}
- Spec did not match any workunits: 'rbd/rbd_mirror_image_replay.sh'
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 6000 -M pool_client.0 image_client.0'
- '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T11:58:37.259856+00:00 smithi003 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - '/home/ubuntu/cephtest/archive/syslog/kern.log:2016-08-26T13:01:52.095902+00:00 smithi003 kernel: [ INFO: possible circular locking dependency detected ]
' in syslog - saw valgrind issues
Re-running failed jobs
#3 Updated by Loïc Dachary over 7 years ago
rgw¶
teuthology-suite -k distro --priority 1000 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
INFO:teuthology.suite:Passed subset=1/5
- fail http://pulpito.ceph.com/loic-2016-08-25_06:41:52-rgw-jewel-backports-distro-basic-smithi
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rgw/s3_bucket_quota.pl'- rgw/multifs/{overrides.yaml clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec.yaml tasks/rgw_bucket_quota.yaml}
- rgw/multifs/{overrides.yaml clusters/fixed-2.yaml frontend/civetweb.yaml fs/btrfs.yaml rgw_pool_type/replicated.yaml tasks/rgw_bucket_quota.yaml}
- rgw/multifs/{overrides.yaml clusters/fixed-2.yaml frontend/apache.yaml fs/xfs.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_bucket_quota.yaml}
- 'mkdir
Re-running the failed tests without https://github.com/ceph/ceph/pull/10832 to verify it is responsible for them
#4 Updated by Loïc Dachary over 7 years ago
rados¶
teuthology-suite -k distro --priority 1000 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
INFO:teuthology.suite:Passed subset=1846/2000
- fail http://pulpito.ceph.com/loic-2016-08-25_06:40:19-rados-jewel-backports-distro-basic-smithi
- known bug test_list_inconsistent_obj: assert len(objs) == 1
- environmental noise smithi048.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False
Re-running failed jobs
- fail http://pulpito.ceph.com/loic-2016-08-29_11:15:32-rados-jewel-backports-distro-basic-smithi
- rados/thrash-erasure-code/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/fastclose.yaml thrashers/default.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml} passed the other two failures are the same known bug as above
#5 Updated by Loïc Dachary over 7 years ago
fs¶
teuthology-suite -k distro --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
INFO:teuthology.suite:Passed subset=3/5
- fail http://pulpito.ceph.com/loic-2016-08-25_06:39:08-fs-jewel-backports-distro-basic-smithi/
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs-java/test.sh'- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3'
- 'mkdir
Re-running failed tests
- fail http://pulpito.ceph.com/loic-2016-08-25_21:09:12-fs-jewel-backports-distro-basic-smithi/
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35a8dd5a4f0ee33de521a80000f5115f0ced8fba TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs-java/test.sh'
- 'mkdir
#6 Updated by Loïc Dachary over 7 years ago
powercycle¶
teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
#7 Updated by Loïc Dachary over 7 years ago
Upgrade¶
teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 1000 machine_types/vps.yaml
- fail http://pulpito.ceph.com/loic-2016-08-25_06:45:48-upgrade:jewel-x-jewel-backports-distro-basic-vps
teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 1000 machine_types/vps.yaml
#8 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#9 Updated by Loïc Dachary over 7 years ago
git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
- + Pull request 9872
- |\
- | + init-radosgw: do not use systemd-run in sysvinit
- + Pull request 10107
- |\
- | + client: kill QuotaTree
- + Pull request 10645
- |\
- | + librbd: journal::Replay no longer holds lock while completing callback
- + Pull request 10650
- |\
- | + librbd: prevent creation of clone from non-primary mirrored image
- + Pull request 10652
- |\
- | + librbd: prevent creation of v2 image ids that are too large
- + Pull request 10757
- |\
- | + mon/osdmonitor: decouple adjust_heartbeat_grace and min_down_reporters
- + Pull request 10758
- |\
- | + ceph-disk: timeout ceph-disk to avoid blocking forever
- + Pull request 10759
- |\
- | + mon/osdmonitor: fix incorrect output of osd df due to osd out
- | + mon/osdmonitor: initialize local variable kb_avail_i
- + Pull request 10784
- |\
- | + librbd: request exclusive lock if current owner cannot execute op
- + Pull request 10796
- |\
- | + rbd: bench io-size should not be larger than image size
- + Pull request 10832
- |\
- | + rgw: Add documentation for RGW default quota
- | + rgw: Do not bother to check against conf values during quota handling
- | + rgw: Let the default quota settings take effect during user creation
- + Pull request 10855
- |\
- | + common: instantiate strict_si_cast<long> not strict_si_cast<int64_t>
- | + common: add int64_t template for strict_si_cast()
- | + common/config: cast OPT_U32 options using uint32_t
- | + Revert common: add int64_t template for strict_si_cast()
- + Pull request 10857
- |\
- | + librbd: fix possible inconsistent state when disabling mirroring
- + Pull request 10860
- |\
- | + mon/PGMonitor: calc the %USED of pool using used/
- | + mon/PGMonitor: mark dump_object_stat_sum() as static
- + Pull request 10861
- |\
- | + crush: When consider rule size use the correct method.
- + Pull request 10862
- |\
- | + rpm: proper packaging of udev directories
- + Pull request 10864
- |\
- | + ObjectCacher: fix last_write check in bh_write_adjacencies()
- + Pull request 10867
- |\
- | + rgw: fix period update --commit return error
- + Pull request 10868
- |\
- | + rgw: fix radosgw daemon core when reopen logs
- + Pull request 10877
- |\
- | + ceph: don't fudge the ctime in stat() unless it's really older than the mtime
- | + client: only skip querying the MDS in _lookup when we have the necessary caps
- | + client: plumb a mask argument into _lookup
- | + client: add mask parameter to _do_lookup
- + Pull request 10883
- |\
- | + PG: use upset rather than up for _update_calc_stats
- | + PG: introduce and maintain upset
- + Pull request 10884
- |\
- | + ceph-disk: do not activate device that is not ready
- + Pull request 10885
- |\
- | + Cleanup: delete find_best_info again
- + Pull request 10886
- |\
- | + ReplicatedPG: call op_applied for submit_log_entries based repops
- + Pull request 10889
- |\
- | + rgw_file: restore local definition of RGWLibFS gc interval
- | + rgw file: remove busy-wait in RGWLibFS::gc()
- + Pull request 10891
- |\
- | + rgw: fix upgrade from old multisite to new multisite configuration
- + Pull request 11056
- |\
- | + librbd: possible deadlock if cluster connection closed after image
- + Pull request 11139
- |\
- | + Add two options to radosgw-admin.rst manpage
- | + radosgw-admin: add --orphan-stale-secs to --help
- | + doc: add --orphan-stale-secs to radosgw-admin
- + Pull request 11158
- |\
- | + Add Install section to systemd rbdmap.service file
- + Pull request 11193
- |\
- | + librados: modify Pipe::connect() to return the error code
- + Pull request 11201
- |\
- | + logrotate: Run as root/ceph
- + Pull request 11231
- |\
- | + log: Log.cc: Assign LOG_INFO priority to syslog calls
- + Pull request 11272
- |\
- | + mon: OSDMonitor: Missing nearfull flag set
- + Pull request 11311
- |\
- | + PG: block writes and scan log for scrub using inclusive upper bound
- | + osd_types,PG: force the map used in _scrub to be sorted correctly
- | + src/osd: relax the requirement that we scrub a whole hash value
- | + hobject: clarify is_snap and has_snapset for max
- + Pull request 11321
- |\
- | + rgw: Do not archive metadata by default
- + Pull request 11322
- |\
- | + rgw: set correct instance on the object
- | + rgw: fix regression with handling double underscore
- + Pull request 11330
- |\
- | + rgw: RGWDataSyncCR fails on errors from RGWListBucketIndexesCR
- + Pull request 11337
- |\
- | + librbd: permit disabling journaling if in corrupt state
- | + librbd: new journal policy to disable initializing the journal
- + Pull request 11342
- |\
- | + doc/radosgw: fix description of response elements 'Part'
- + Pull request 11367
- |\
- | + rgw: forward input data when forwarding set_bucket_version to master
- | + rgw: set args when rebuilding req_info
- + Pull request 11407
- |\
- | + os/filestore/FileJournal: fail out if FileJournal is not block device or regular file
- + Pull request 11408
- |\
- | + cephx: Fix multiple segfaults due to attempts to encrypt or decrypt an empty secret and a null CryptoKeyHandler
- + Pull request 11409
- |\
- | + common: only call crypto::init once per CephContext
- + Pull request 11410
- |\
- | + crush/CrushCompiler: error out as long as parse fails
- + Pull request 11411
- |\
- | + tests: populate /dev/disk/by-partuuid for scsi_debug
- + Pull request 11412
- |\
- | + mds: catch duplicates in DamageTable
- + Pull request 11413
- |\
- | + mds: handle blacklisting during journal recovery
- | + mds: use a random nonce in Messenger
- + Pull request 11414
- |\
- | + client: fix segment fault in Client::_invalidate_kernel_dcache().
- + Pull request 11415
- |\
- | + mds: remove fail-safe queueing replay request
- + Pull request 11416
- |\
- | + client: fix readdir vs fragmentation race
- + Pull request 11417
- |\
- | + ceph-create-keys: fix existing-but-different case
- + Pull request 11418
- |\
- | + mds: log path with CDir damage messages
- | + mds: s/used_ions/used_inos/
- + Pull request 11419
- |\
- | + client: properly set inode number of created inode in replay request
- + Pull request 11420
- |\
- | + librbd: ignore notify errors on missing image header
- + Pull request 11431
- |\
- | + build: include more files in make dist tarball
- + Pull request 11433
- + journal: ensure in-flight ops are complete destroying journaler
- + journal: complete action only after notification completed
- + qa/workunits: reduce amount of client IO for rbd-mirror stress
- + rbd-mirror: potential race allocating tag during shut down
- + librbd: ignore attempts to request lock if already lock owner
- + journal: clean up object recorder closed/overflow callback
- + journal: delay moving overflowed buffers until quiesced
- + journal: use reverse iterator search to find flush record
- + journal: avoid holding lock while sending journal append
- + rbd-mirror: wait for in-flight event commit notifications
- + journal: fix AsyncOpTracker compiler error
- + common: move AsyncOpTracker to common library
- + librbd: support deleting image by id instead of name
- + rbd-mirror: use pool id + global image id as deletion primary key
- + rbd-mirror: concurrent access of event might result in heap corruption
- + rbd-mirror: test: Fixed timeout problem in rbd_mirror_stress.sh
- + rbd: Fix race between journal flush and append events
- + journal: make librados call async in ObjectRecorder
- + journal: increase concurrency of journal recorder
- + journal: send update notification when tag allocated
- + librbd: new journal listener event for force promotion
- + librbd: helper class for quiescing in-flight async ops
- + librbd: unify journal event callbacks into single interface
- + journal: expose ability to retrieve partial tag list
- + qa/workunits/rbd: fix remove mirrored image race conditions
- + qa/workunits/rbd: new mirroring forced promotion test
- + test/rbd: fix possible mock journal race conditions
- + qa/workunits/rbd: before removing image make sure it is not bootstrapped
- + librbd: fix journal create compilation error
- + test: fixed unused function warnings in unittest_rbd_mirror
- + rbd-mirror: prevent syncing to non-primary image after failover
- + rbd-mirror: demote/promote in same cluster results in split-brain
- + librbd: normalize journal promotion/demotion events
- + librbd: block RPC requests when demoting the image
- + librbd: separate journal::TagData predecessor data into new struct
- + rbd-mirror: include tag tid in bootstrap debug log messages
- + qa/workunits/rbd: demote/promote image on same cluster
- + rbd-mirror: option to automatically resync after journal client disconnect
- + rbd-mirror: stop replay when client is disconnected
- + rbd-mirror: resync was possible only when image replayer start had succeeded
- + rbd-mirror: decode_client_meta should return false on error
- + rbd: new command to disconnect journal client
- + librbd: optionally flag laggy journal clients disconnected
- + journal: allow to trim journal for laggy clients
- + cls/journal: add async client_update_state method
#10 Updated by Loïc Dachary over 7 years ago
rbd¶
teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-13_09:17:31-rbd-jewel-backports---basic-smithi/
- new bug FAILED assert 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0'
- rbd/qemu/{cache/writeback.yaml cachepool/ec-cache.yaml clusters/{fixed-3.yaml openstack.yaml} features/journaling.yaml fs/xfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}
- rbd/qemu/{cache/writeback.yaml cachepool/small.yaml clusters/{fixed-3.yaml openstack.yaml} features/journaling.yaml fs/xfs.yaml msgr-failures/few.yaml workloads/qemu_xfstests.yaml}
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=94a52962cdb0eb3d2f2eba11daa0e14ae349734e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/qemu-iotests.sh' - incoming backport jewel: ImageWatcher: use after free within C_UnwatchAndFlush saw valgrind issues
- new bug FAILED assert 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0'
Re-run failed tests
- fail http://pulpito.ceph.com/loic-2016-10-14_06:48:18-rbd-jewel-backports---basic-smithi
- new bug FAILED assert 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0'
- incoming backport jewel: ImageWatcher: use after free within C_UnwatchAndFlush saw valgrind issues
The dead jobs are because of a known bug rbd-nbd IO hang
#11 Updated by Loïc Dachary over 7 years ago
rgw¶
teuthology-suite -k distro --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-13_09:20:07-rgw-jewel-backports-distro-basic-smithi
- HTTPConnectionPool(host='smithi006.front.sepia.ceph.com', port=8000): Max retries exceeded with url: /metadata/incremental (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe8c9d343d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
- HTTPConnectionPool(host='smithi078.front.sepia.ceph.com', port=8000): Max retries exceeded with url: /metadata/incremental (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fad43d25a50>: Failed to establish a new connection: [Errno 111] Connection refused',))
Re-running failed tests
#12 Updated by Loïc Dachary over 7 years ago
rados¶
teuthology-suite -k distro --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-13_09:21:20-rados-jewel-backports-distro-basic-smithi
- **
- , 'smithi099.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
- , 'smithi052.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}, 'smithi113.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
- , 'smithi100.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
- "2016-10-13 11:39:40.845873 osd.5 172.21.15.5:6808/4055 10 : cluster [ERR] 7.0 scrub 0 missing, 2 inconsistent objects" in cluster log
Re-run failed tests
#13 Updated by Loïc Dachary over 7 years ago
fs¶
teuthology-suite -k distro --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-13_09:23:01-fs-jewel-backports-distro-basic-smithi
- }
- }
- }
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=94a52962cdb0eb3d2f2eba11daa0e14ae349734e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs/test.sh'- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/valgrind.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=94a52962cdb0eb3d2f2eba11daa0e14ae349734e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/suites/blogbench.sh' - }
- , 'smithi102.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
- }
- }
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=94a52962cdb0eb3d2f2eba11daa0e14ae349734e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs-java/test.sh'- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- }
- , 'smithi047.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
- }
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3'
Re-run failed tests
- fail http://pulpito.ceph.com/loic-2016-10-13_17:04:37-fs-jewel-backports-distro-basic-smithi/
- known bug LibCephFS.InterProcessLocking failing on master
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/valgrind.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}
- known bug LibCephFS.InterProcessLocking failing on master
#14 Updated by Loïc Dachary over 7 years ago
powercycle¶
teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
#15 Updated by Loïc Dachary over 7 years ago
Upgrade¶
teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
- dead http://pulpito.ceph.com/loic-2016-10-13_09:27:45-upgrade:jewel-x-jewel-backports-distro-basic-vps
Re-run dead jobs
- fail http://pulpito.ceph.com/loic-2016-10-14_06:56:49-upgrade:jewel-x-jewel-backports-distro-basic-vps
- new bug jewel: upgrade TestExclusiveLock.test_follower_flatten ERROR test_librbd_python.sh
teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
- fail http://pulpito.ceph.com/loic-2016-10-13_09:28:11-upgrade:hammer-x-jewel-backports-distro-basic-vps
- }
- , 'vpm057.front.sepia.ceph.com': {'msg': 'One or more items failed', 'failed': True, 'changed': False}}
Re-run failed jobs
#16 Updated by Loïc Dachary over 7 years ago
git --no-pager log --format='%H %s' --graph ceph/jewel..jewel-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'
- + Pull request 10757
- |\
- | + mon/osdmonitor: decouple adjust_heartbeat_grace and min_down_reporters
- + Pull request 10759
- |\
- | + mon/osdmonitor: fix incorrect output of osd df due to osd out
- | + mon/osdmonitor: initialize local variable kb_avail_i
- + Pull request 10860
- + Pull request 10862
- + Pull request 10884
- + Pull request 10886
- + Pull request 11126
- + Pull request 11158
- + Pull request 11201
- + Pull request 11321
- |\
- | + rgw: Do not archive metadata by default
- + Pull request 11407
- + Pull request 11408
- + Pull request 11411
- + Pull request 11431
- + Pull request 11459
- + Pull request 11460
- + Pull request 11461
- + Pull request 11462
- + Pull request 11463
- + Pull request 11464
- + Pull request 11466
- + Pull request 11469
- + Pull request 11470
- |\
- | + rgw:bucket check remove multipart prefix
- + Pull request 11471
- + Pull request 11472
- |\
- | + rgw: fix versioned object IO error
- + Pull request 11473
- + Pull request 11474
- + Pull request 11475
- + Pull request 11476
- |\
- | + rgw: RGWCoroutinesManager::run returns status of last cr
- + Pull request 11477
- |\
- | + rgw: fix for assertion in RGWMetaSyncCR
- + Pull request 11478
- |\
- | + Allow building RGW with LDAP disabled
- + Pull request 11492
- + Pull request 11508
- |\
- | + utime.h: fix timezone issue in round_to_* funcs.
- + Pull request 11519
#17 Updated by Loïc Dachary over 7 years ago
rbd¶
teuthology-suite --priority 101 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-18_17:19:39-rbd-jewel-backports---basic-smithi/
- 'CEPH_REF=master CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram
v -/home/ubuntu/cephtest/archive/cram.client.0/.t'* - 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/qemu-iotests.sh'
- 'CEPH_REF=master CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram
Re-running failed jobs
The dead jobs are because of a known bug rbd-nbd IO hang
#18 Updated by Loïc Dachary over 7 years ago
rgw¶
teuthology-suite -k distro --priority 101 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-19_06:43:51-rgw-jewel-backports-distro-basic-smithi
- HTTPConnectionPool(host='smithi046.front.sepia.ceph.com', port=8000): Max retries exceeded with url: /metadata/incremental (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa8c0fbb490>: Failed to establish a new connection: [Errno 111] Connection refused',))
- HTTPConnectionPool(host='smithi114.front.sepia.ceph.com', port=8000): Max retries exceeded with url: /metadata/incremental (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f994d1d8410>: Failed to establish a new connection: [Errno 111] Connection refused',))
Re-running failed jobs
#19 Updated by Loïc Dachary over 7 years ago
rados¶
teuthology-suite -k distro --priority 101 --suite rados --subset $(expr $RANDOM % 2000)/2000 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-19_06:45:42-rados-jewel-backports-distro-basic-smithi
- known bug test_list_inconsistent_obj: assert len(objs) == 1
- known bug bad rados_buffer_free from librados python bindings 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'
Re-running failed jobs
- pass http://pulpito.ceph.com/loic-2016-10-19_21:00:55-rados-jewel-backports-distro-basic-smithi
- known bug test_list_inconsistent_obj: assert len(objs) == 1
- known bug bad rados_buffer_free from librados python bindings 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'
Running rados/singleton/{rados.yaml all/cephtool.yaml fs/xfs.yaml msgr/random.yaml msgr-failures/many.yaml} against jewel to check if it shows
Running rados/singleton/{rados.yaml all/cephtool.yaml fs/xfs.yaml msgr/random.yaml msgr-failures/many.yaml} against jewel-backports 10 times to see how frequently it shows up
#20 Updated by Loïc Dachary over 7 years ago
fs¶
teuthology-suite -k distro --priority 101 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch jewel --email loic@dachary.org --ceph jewel-backports --machine-type smithi
- fail http://pulpito.ceph.com/loic-2016-10-19_06:48:12-fs-jewel-backports-distro-basic-smithi
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs/test.sh'- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/valgrind.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs-java/test.sh'- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_java.yaml}
- 'mkdir
Re-running failed jobs
- fail http://pulpito.ceph.com/loic-2016-10-19_21:02:41-fs-jewel-backports-distro-basic-smithi
- known bug LibCephFS.InterProcessLocking failing on master
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/no.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/valgrind.yaml}
- fs/basic/{clusters/fixed-2-ucephfs.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
- fs/verify/{clusters/fixed-2-ucephfs.yaml debug/{mds_client.yaml mon.yaml} dirfrag/frag_enable.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbe23a89240e5c5ea7ed0b0c8ca3783a2172b6e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs-java/test.sh'
- known bug LibCephFS.InterProcessLocking failing on master
#21 Updated by Loïc Dachary over 7 years ago
powercycle¶
teuthology-suite -l2 -v -c jewel-backports -k testing -m smithi -s powercycle -p 1000 --email loic@dachary.org
#22 Updated by Loïc Dachary over 7 years ago
Upgrade¶
teuthology-suite -k distro --verbose --suite upgrade/jewel-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
- fail http://pulpito.ceph.com/loic-2016-10-19_06:50:27-upgrade:jewel-x-jewel-backports-distro-basic-vps
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=jewel TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'
- 'mkdir
Re-running failed jobs
- pass http://pulpito.ceph.com/loic-2016-10-19_21:05:03-upgrade:jewel-x-jewel-backports-distro-basic-vps
teuthology-suite -k distro --verbose --suite upgrade/hammer-x --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
- fail http://pulpito.ceph.com/loic-2016-10-19_06:51:10-upgrade:hammer-x-jewel-backports-distro-basic-vps
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2'
Re-running failed jobs
#23 Updated by Loïc Dachary over 7 years ago
ceph-disk¶
teuthology-suite -k distro --verbose --suite ceph-disk --suite-branch jewel --ceph jewel-backports --machine-type vps --priority 101 machine_types/vps.yaml
#24 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#25 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#26 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#27 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#28 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#29 Updated by Nathan Cutler over 7 years ago
- Description updated (diff)
#30 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#31 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#32 Updated by Yuri Weinstein over 7 years ago
QE VALIDATION (STARTED 11/7/16)¶
(Note: PASSED / FAILED - indicates "TEST IS IN PROGRESS")
re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v10.2.4_QE_validation_notes
command line CEPH_BRANCH=jewel; MACHINE_NAME=vps; teuthology-suite -v -S $CEPH_BRANCH -m $MACHINE_NAME -k distro -s rados -e $CEPH_QA_EMAIL --suite-branch jewel
kcephfs | http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:06:21-kcephfs-jewel-testing-basic-smithi/ | PASSED |
rest | http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:09:43-rest-jewel-distro-basic-smithi/ | PASSED |
hadoop | http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:10:15-hadoop-jewel-distro-basic-smithi/ | PASSED |
ceph-disk | http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-07_17:12:50-ceph-disk-jewel-distro-basic-vps/ | PASSED |
upgrade/infernalis-x (jewel) | http://pulpito.front.sepia.ceph.com:80/yuriw-2016-11-14_16:21:41-upgrade:infernalis-x-jewel-distro-basic-vps/ | UNSUPPORTED #17683, #17874 |
upgrade/jewel-x/point-to-point-x | http://pulpito.ceph.com/yuriw-2016-11-16_15:07:00-upgrade:jewel-x:point-to-point-x-jewel-distro-basic-smithi/ | PASSED |
PASSED / FAILED | ||
#33 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#34 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#35 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#36 Updated by Nathan Cutler over 7 years ago
- Description updated (diff)
Added release notes entry as per https://github.com/ceph/ceph/pull/11679#issuecomment-259780456
#37 Updated by Abhishek Lekshmanan over 7 years ago
Running a suite for the last 2 prs which will make it to 10.2.4
This is the current jewel branch as of 427f357f0eed32c9ce17590ae9303a94e8b710e7 + the prs
https://github.com/ceph/ceph/pull/12001
https://github.com/ceph/ceph/pull/12167
https://github.com/ceph/ceph/pull/12067
CEPH_BRANCH=wip-jewel-10-2-4; MACHINE_NAME=smithi; SUITE_BRANCH=jewel; teuthology-suite --dry-run -v -c $CEPH_BRANCH -m $MACHINE_NAME --suite-branch jewel -k distro -s rados --subset 35/50 -e ceph-qa@ceph.com -p 100
- saw valgrind issues
- rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/simple.yaml msgr-failures/few.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml}
- rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/async.yaml msgr-failures/few.yaml tasks/mon_recovery.yaml validater/valgrind.yaml}
- '/home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite'
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-106-gbfda134-1xenial ceph-mds=0.94.9-106-gbfda134-1xenial ceph-common=0.94.9-106-gbfda134-1xenial ceph-fuse=0.94.9-106-gbfda134-1xenial ceph-test=0.94.9-106-gbfda134-1xenial radosgw=0.94.9-106-gbfda134-1xenial python-ceph=0.94.9-106-gbfda134-1xenial libcephfs1=0.94.9-106-gbfda134-1xenial libcephfs-java=0.94.9-106-gbfda134-1xenial libcephfs-jni=0.94.9-106-gbfda134-1xenial librados2=0.94.9-106-gbfda134-1xenial librbd1=0.94.9-106-gbfda134-1xenial rbd-fuse=0.94.9-106-gbfda134-1xenial'
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
- rados/singleton-nomsgr/{rados.yaml all/lfn-upgrade-infernalis.yaml}
- rados/singleton/{rados.yaml all/lost-unfound-upgrade.yaml fs/xfs.yaml msgr/async.yaml msgr-failures/many.yaml}
- rados/singleton/{rados.yaml all/lost-unfound-delete-upgrade.yaml fs/xfs.yaml msgr/random.yaml msgr-failures/many.yaml}
- rados/singleton/{rados.yaml all/ec-lost-unfound-upgrade.yaml fs/xfs.yaml msgr/simple.yaml msgr-failures/many.yaml}
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.80.8/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.67.10/version
- "rados/singleton-nomsgr/{rados.yaml all/13234.yaml}"::http://pulpito.ceph.com/abhi-2016-11-29_10:22:25-rados-wip-jewel-10-2-4-distro-basic-smithi/584907
Re-running failed jobs:
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-11-30_09:07:29-rados-wip-jewel-10-2-4-distro-basic-smithi/
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.80.8/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.67.10/version
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-106-gbfda134-1xenial ceph-mds=0.94.9-106-gbfda134-1xenial ceph-common=0.94.9-106-gbfda134-1xenial ceph-fuse=0.94.9-106-gbfda134-1xenial ceph-test=0.94.9-106-gbfda134-1xenial radosgw=0.94.9-106-gbfda134-1xenial python-ceph=0.94.9-106-gbfda134-1xenial libcephfs1=0.94.9-106-gbfda134-1xenial libcephfs-java=0.94.9-106-gbfda134-1xenial libcephfs-jni=0.94.9-106-gbfda134-1xenial librados2=0.94.9-106-gbfda134-1xenial librbd1=0.94.9-106-gbfda134-1xenial rbd-fuse=0.94.9-106-gbfda134-1xenial'
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
- saw valgrind issues
- rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/simple.yaml msgr-failures/few.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml}
- rados/verify/{rados.yaml 1thrash/none.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/async.yaml msgr-failures/few.yaml tasks/mon_recovery.yaml validater/valgrind.yaml}
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-11-30_10:12:21-rados-wip-jewel-10-2-4-distro-basic-smithi/
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.80.8/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/v0.67.10/version
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-106-gbfda134-1xenial ceph-mds=0.94.9-106-gbfda134-1xenial ceph-common=0.94.9-106-gbfda134-1xenial ceph-fuse=0.94.9-106-gbfda134-1xenial ceph-test=0.94.9-106-gbfda134-1xenial radosgw=0.94.9-106-gbfda134-1xenial python-ceph=0.94.9-106-gbfda134-1xenial libcephfs1=0.94.9-106-gbfda134-1xenial libcephfs-java=0.94.9-106-gbfda134-1xenial libcephfs-jni=0.94.9-106-gbfda134-1xenial librados2=0.94.9-106-gbfda134-1xenial librbd1=0.94.9-106-gbfda134-1xenial rbd-fuse=0.94.9-106-gbfda134-1xenial'
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
- saw valgrind issues
Further runs are still failing with similar errors, reported this issue as http://tracker.ceph.com/issues/18089
Rerunning failed tests- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-11-30_19:56:24-rados-wip-jewel-10-2-4-distro-basic-smithi/
(dead + valgrind issues)
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-12-01_14:37:49-rados-wip-jewel-10-2-4-distro-basic-smithi/
valgrind exception + dead jobs, - saw valgrind issues
- DEAD http://pulpito.front.sepia.ceph.com/abhi-2016-12-02_10:06:39-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/
The jobs were dying due to a suite still referencing trusty instead of rados, this was fixed in https://github.com/ceph/ceph-qa-suite/pull/1297 - PASS http://pulpito.front.sepia.ceph.com/abhi-2016-12-05_10:25:26-rados-wip-jewel-10-2-4-rc-distro-basic-smithi/
CEPH_BRANCH=wip-jewel-10-2-4; MACHINE_NAME=smithi; SUITE_BRANCH=jewel; teuthology-suite -v -c $CEPH_BRANCH -m $MACHINE_NAME --suite-branch jewel -k distro -s upgrade/client-upgrade -e ceph-qa@ceph.com -p 100
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-106-gbfda134-1xenial ceph-mds=0.94.9-106-gbfda134-1xenial ceph-common=0.94.9-106-gbfda134-1xenial ceph-fuse=0.94.9-106-gbfda134-1xenial ceph-test=0.94.9-106-gbfda134-1xenial radosgw=0.94.9-106-gbfda134-1xenial python-ceph=0.94.9-106-gbfda134-1xenial libcephfs1=0.94.9-106-gbfda134-1xenial libcephfs-java=0.94.9-106-gbfda134-1xenial libcephfs-jni=0.94.9-106-gbfda134-1xenial librados2=0.94.9-106-gbfda134-1xenial librbd1=0.94.9-106-gbfda134-1xenial rbd-fuse=0.94.9-106-gbfda134-1xenial'
- upgrade:client-upgrade/hammer-client-x/basic/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_cli_import_export.yaml}
- upgrade:client-upgrade/hammer-client-x/rbd/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_notification_tests.yaml}
- upgrade:client-upgrade/hammer-client-x/basic/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_api_tests.yaml}
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/firefly/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
- upgrade:client-upgrade/infernalis-client-x/basic/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_cli_import_export.yaml}
- upgrade:client-upgrade/infernalis-client-x/rbd/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_notification_tests.yaml}
- upgrade:client-upgrade/infernalis-client-x/basic/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_api_tests.yaml}
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-11-30_09:58:50-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-106-gbfda134-1xenial ceph-mds=0.94.9-106-gbfda134-1xenial ceph-common=0.94.9-106-gbfda134-1xenial ceph-fuse=0.94.9-106-gbfda134-1xenial ceph-test=0.94.9-106-gbfda134-1xenial radosgw=0.94.9-106-gbfda134-1xenial python-ceph=0.94.9-106-gbfda134-1xenial libcephfs1=0.94.9-106-gbfda134-1xenial libcephfs-java=0.94.9-106-gbfda134-1xenial libcephfs-jni=0.94.9-106-gbfda134-1xenial librados2=0.94.9-106-gbfda134-1xenial librbd1=0.94.9-106-gbfda134-1xenial rbd-fuse=0.94.9-106-gbfda134-1xenial'
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/firefly/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-12-01_10:17:39-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
- u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.94.9-110-g12d6781-1xenial ceph-mds=0.94.9-110-g12d6781-1xenial ceph-common=0.94.9-110-g12d6781-1xenial ceph-fuse=0.94.9-110-g12d6781-1xenial ceph-test=0.94.9-110-g12d6781-1xenial radosgw=0.94.9-110-g12d6781-1xenial python-ceph=0.94.9-110-g12d6781-1xenial libcephfs1=0.94.9-110-g12d6781-1xenial libcephfs-java=0.94.9-110-g12d6781-1xenial libcephfs-jni=0.94.9-110-g12d6781-1xenial librados2=0.94.9-110-g12d6781-1xenial librbd1=0.94.9-110-g12d6781-1xenial rbd-fuse=0.94.9-110-g12d6781-1xenial'
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/firefly/version
- Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-xenial-x86_64-basic/ref/infernalis/version
Re ran after fixes were made to https://github.com/ceph/ceph-qa-suite/pull/1292 and merged
- FAILED http://pulpito.front.sepia.ceph.com/abhi-2016-12-01_15:35:58-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'
- upgrade:client-upgrade/infernalis-client-x/basic/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_cli_import_export.yaml distros/centos_7.2.yaml}
- upgrade:client-upgrade/infernalis-client-x/basic/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_api_tests.yaml distros/centos_7.2.yaml}
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring'
- upgrade:client-upgrade/infernalis-client-x/rbd/{0-cluster/start.yaml 1-install/infernalis-client-x.yaml 2-workload/rbd_notification_tests.yaml distros/centos_7.2.yaml}
- upgrade:client-upgrade/hammer-client-x/rbd/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_notification_tests.yaml distros/centos_7.2.yaml}
- 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'
failures on ceph-create-keys
rerunning to avoid env noise as
- FAILED http://pulpito.ceph.com/abhi-2016-12-01_22:40:25-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/, failed with same reason
- FAILED http://pulpito.ceph.com/abhi-2016-12-02_09:45:37-upgrade:client-upgrade-wip-jewel-10-2-4-distro-basic-smithi/ but only for the I-J upgrade
- 'mkdir
p -/home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=infernalis TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_FEATURES=13 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_api.sh'
- 'mkdir
rgw
This is the jewel branch above + the pr https://github.com/ceph/ceph/pull/12207
teuthology-suite -k distro --priority 100 --suite rgw --subset $(expr $RANDOM % 5)/5 --suite-branch jewel-avoid-xenial --email abhishek@suse.com --ceph wip-jewel-10-2-4-rc --machine-type smithi
- FAILEd http://pulpito.ceph.com/abhi-2016-12-01_21:56:20-rgw-wip-jewel-10-2-4-rc-distro-basic-smithi/
but this was because the branch referring was already merged back to jewel, rescheduled as - FAILED http://pulpito.ceph.com/abhi-2016-12-01_22:19:07-rgw-wip-jewel-10-2-4-rc-distro-basic-smithi/ (valgrind issue, ignore?)
#38 Updated by Sergey Jerusalimov over 7 years ago
Gay's please tell about https://www.spinics.net/lists/ceph-devel/msg33968.html
#39 Updated by Sergey Jerusalimov over 7 years ago
Sergey Jerusalimov wrote:
Guys's please tell about https://www.spinics.net/lists/ceph-devel/msg33968.html
#40 Updated by Nathan Cutler over 7 years ago
@Sergey: That issue will be fixed in 10.2.5, see #18207
#41 Updated by Loïc Dachary over 7 years ago
- Status changed from In Progress to Resolved