Ceph » Stable releases

http://pulpito.ceph.com/loic-2015-08-15_22:14:11-upgrade:hammer-hammer-backports---basic-multi/

Sage: every single test was stuck, and on the 3 I checked
all mon's were stuck in leveldb from scrub using 100% cpu:

#0  0x00007fa14bd164a0 in 
std::string::_Rep::_M_destroy(std::allocator<char> const&) () from 
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00000000008a4c45 in _M_dispose (__a=..., this=<optimized out>) at 
/usr/include/c++/4.8/bits/basic_string.h:249
#2  ~basic_string (this=0x7fa145fe1880, __in_chrg=<optimized out>) at 
/usr/include/c++/4.8/bits/basic_string.h:539
#3  LevelDBStore::split_key (in=..., prefix=prefix@entry=0x0, 
key=key@entry=0x7fa145fe1990) at os/LevelDBStore.cc:230
#4  0x00000000008a73d6 in LevelDBStore::LevelDBWholeSpaceIteratorImpl::key 
(this=<optimized out>) at os/LevelDBStore.h:271
#5  0x000000000059a86e in KeyValueDB::IteratorImpl::key (this=<optimized 
out>) at ./os/KeyValueDB.h:146
#6  0x00000000008a5fdf in LevelDBStore::get (this=0x361b080, prefix=..., 
keys=..., out=0x7fa145fe1b60) at os/LevelDBStore.cc:195
#7  0x000000000059fc0a in MonitorDBStore::get (this=0x361b1e0, prefix=..., 
key=..., bl=...) at mon/MonitorDBStore.h:499
#8  0x00000000005b268a in Monitor::_scrub (this=this@entry=0x3792000, 
r=0x360d358) at mon/Monitor.cc:4272
#9  0x00000000005c2531 in Monitor::scrub (this=this@entry=0x3792000) at 
mon/Monitor.cc:4217
#10 0x00000000005cc6b7 in Monitor::handle_command 
(this=this@entry=0x3792000, m=m@entry=0x3ad3a00) at mon/Monitor.cc:2711
#11 0x00000000005cefa9 in Monitor::dispatch (this=this@entry=0x3792000, 
s=s@entry=0x424ae00, m=m@entry=0x3ad3a00, 
src_is_mon=src_is_mon@entry=false) at mon/Monitor.cc:3457
#12 0x00000000005cfc26 in Monitor::_ms_dispatch 
(this=this@entry=0x3792000, m=m@entry=0x3ad3a00) at mon/Monitor.cc:3376
#13 0x00000000005cea35 in Monitor::handle_forward 
(this=this@entry=0x3792000, m=m@entry=0x3c0f080) at mon/Monitor.cc:3068
#14 0x00000000005cf66d in Monitor::dispatch (this=this@entry=0x3792000, 
s=s@entry=0x3636e00, m=m@entry=0x3c0f080, 
src_is_mon=src_is_mon@entry=true) at mon/Monitor.cc:3589
#15 0x00000000005cfc26 in Monitor::_ms_dispatch 
(this=this@entry=0x3792000, m=m@entry=0x3c0f080) at mon/Monitor.cc:3376
#16 0x00000000005ede23 in Monitor::ms_dispatch (this=0x3792000, 
m=0x3c0f080) at mon/Monitor.h:833
#17 0x00000000009277c9 in ms_deliver_dispatch (m=0x3c0f080, 
this=0x37a2700) at ./msg/Messenger.h:567
#18 DispatchQueue::entry (this=0x37a28c8) at 
msg/simple/DispatchQueue.cc:185
#19 0x00000000007c7fcd in DispatchQueue::DispatchThread::entry 
(this=<optimized out>) at msg/simple/DispatchQueue.h:103
#20 0x00007fa14cf0a182 in start_thread (arg=0x7fa145fe4700) at 
pthread_create.c:312
#21 0x00007fa14b474fbd in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:111

• https://github.com/ceph/ceph/pull/5361

• https://github.com/ceph/ceph/pull/5359
• https://github.com/ceph/ceph/pull/5377

• https://github.com/ceph/ceph/pull/5376
• https://github.com/ceph/ceph/pull/5372
• https://github.com/ceph/ceph/pull/5370

• https://github.com/ceph/ceph/pull/5471

The stack trace appears to get stuck while dealloc'ing a string. I don't really know how this can happen.

Joao Luis wrote:

The stack trace appears to get stuck while dealloc'ing a string. I don't really know how this can happen.

The stack traces vary slightly, they're all in get_key() but with various other bits after that:

Thread 31 (Thread 0x7f3c24479700 (LWP 12869)):
#0  LevelDBStore::get (this=0x41231e0, prefix=..., keys=..., out=0x7f3c24477e70) at /usr/include/c++/4.8/bits/stl_set.h:299
#1  0x000000000059fc0a in MonitorDBStore::get (this=0x4123340, prefix=..., key=..., bl=...) at mon/MonitorDBStore.h:499
#2  0x00000000005b268a in Monitor::_scrub (this=this@entry=0x4282000, r=r@entry=0x4897848) at mon/Monitor.cc:4272

Thread 33 (Thread 0x7fdad6593700 (LWP 12898)):
#0  LevelDBStore::split_key (in=..., prefix=prefix@entry=0x7fdad6590890, key=key@entry=0x7fdad65908a0) at os/LevelDBStore.cc:226
#1  0x00000000008a7462 in LevelDBStore::LevelDBWholeSpaceIteratorImpl::raw_key (this=<optimized out>) at os/LevelDBStore.h:276
#2  0x000000000059aaaf in KeyValueDB::IteratorImpl::valid (this=0x5139e20) at ./os/KeyValueDB.h:132
#3  0x00000000008a5fc0 in LevelDBStore::get (this=0x50b8f20, prefix=..., keys=..., out=0x7fdad6590b60) at os/LevelDBStore.cc:195
#4  0x000000000059fc0a in MonitorDBStore::get (this=0x50b9080, prefix=..., key=..., bl=...) at mon/MonitorDBStore.h:499
#5  0x00000000005b268a in Monitor::_scrub (this=this@entry=0x521e000, r=0x50aa398) at mon/Monitor.cc:4272

Thread 31 (Thread 0x7f93f0e82700 (LWP 12899)):
#0  0x00007f93f658be70 in std::string::compare(std::string const&) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00000000005b2875 in operator< <char, std::char_traits<char>, std::allocator<char> > (__rhs=..., __lhs=...) at /usr/include/c++/4.8/bits/basic_string.h:2571
#2  operator() (this=<optimized out>, __y=..., __x=...) at /usr/include/c++/4.8/bits/stl_function.h:235
#3  operator[] (__k=..., this=0x4b005f8) at /usr/include/c++/4.8/bits/stl_map.h:463
#4  Monitor::_scrub (this=this@entry=0x465a000, r=r@entry=0x4b005c8) at mon/Monitor.cc:4274

Thread 33 (Thread 0x7f0194204700 (LWP 14872)):
#0  ~basic_string (this=0x7f0194202b90, __in_chrg=<optimized out>) at /usr/include/c++/4.8/bits/basic_string.h:539
#1  raw (l=1652, this=0x3aea0b0) at common/buffer.cc:135
#2  raw_char (l=1652, this=0x3aea0b0) at common/buffer.cc:483
#3  ceph::buffer::copy (c=0x7f018bad1012 "\001\002\002\202\002", len=1652) at common/buffer.cc:589
#4  0x000000000084f888 in ceph::buffer::ptr::ptr (this=0x7f0194202be0, d=<optimized out>, l=<optimized out>) at common/buffer.cc:651
#5  0x00000000008a59d4 in LevelDBStore::to_bufferlist (in=...) at os/LevelDBStore.cc:215
#6  0x00000000008a788e in LevelDBStore::LevelDBWholeSpaceIteratorImpl::value (this=<optimized out>) at os/LevelDBStore.h:280
#7  0x000000000059a88e in KeyValueDB::IteratorImpl::value (this=<optimized out>) at ./os/KeyValueDB.h:149
#8  0x00000000008a6053 in LevelDBStore::get (this=0x38c8f20, prefix=..., keys=..., out=0x7f0194202e70) at os/LevelDBStore.cc:196
#9  0x000000000059fc0a in MonitorDBStore::get (this=0x38c9080, prefix=..., key=..., bl=...) at mon/MonitorDBStore.h:499
#10 0x00000000005b268a in Monitor::_scrub (this=this@entry=0x3a6c000, r=r@entry=0x4067c48) at mon/Monitor.cc:4272

Thread 33 (Thread 0x7fcb0a153700 (LWP 14871)):
#0  0x00000000008a73c3 in LevelDBStore::LevelDBWholeSpaceIteratorImpl::key (this=0x3b807d0) at os/LevelDBStore.h:271
#1  0x000000000059a86e in KeyValueDB::IteratorImpl::key (this=<optimized out>) at ./os/KeyValueDB.h:146
#2  0x00000000008a5fdf in LevelDBStore::get (this=0x3ba4f20, prefix=..., keys=..., out=0x7fcb0a151e70) at os/LevelDBStore.cc:195
#3  0x000000000059fc0a in MonitorDBStore::get (this=0x3ba5080, prefix=..., key=..., bl=...) at mon/MonitorDBStore.h:499
#4  0x00000000005b268a in Monitor::_scrub (this=this@entry=0x3d48000, r=r@entry=0x3fb8388) at mon/Monitor.cc:4272

and all are at 100% cpu. pretty clearly busy looping in get()!

Probably something in the caller is different on hammer vs master? :/

In any case, I'd drop that commit!

$ git rev-parse ceph/hammer-backports
63c3d50ace54238418cec1d5ebb5a32364058cad

git --no-pager log --format='%H %s' --graph ceph/hammer..ceph/hammer-backports | perl -p -e 's/"/ /g; if (/\w+\s+Merge pull request #(\d+)/) { s|\w+\s+Merge pull request #(\d+).*|"Pull request $1":https://github.com/ceph/ceph/pull/$1|; } else { s|(\w+)\s+(.*)|"$2":https://github.com/ceph/ceph/commit/$1|; } s/\*/+/; s/^/* /;'

• + Pull request 5276
• |\
• | + mon: test the crush ruleset when creating a pool
• | + erasure-code: set max_size to chunk_count() instead of 20 for shec
• | + vstart.sh: set PATH to include pwd
• + Pull request 5367
• |\
• | + os/chain_xattr: handle read on chnk-aligned xattr
• + Pull request 5373
• |\
• | + osd: pg_interval_t::check_new_interval should not rely on pool.min_size to determine if the PG was active
• | + osd: Move IsRecoverablePredicate/IsReadablePredicate to osd_types.h
• + Pull request 5377
• |\
• | + mon/PGMonitor: bug fix pg monitor get crush rule
• + Pull request 5381
• |\
• | + Client: check dir is still complete after dropping locks in _readdir_cache_cb
• + Pull request 5382
• |\
• | + auth: check return value of keyring->get_secret
• + Pull request 5383
• |\
• | + rest_bench: bucketname is not mandatory as we have a default name
• | + rest_bench: drain the work queue to fix a crash Fixes: #3896 Signed-off-by: huangjun <hjwsm1989@gmail.com>
• + Pull request 5498
• |\
• | + rgw: set http status in civetweb
• | + civetweb: update submodule to support setting of http status
• + Pull request 5527
• |\
• | + osd/OSDMap: handle incrementals that modify+del pool
• + Pull request 5697
• |\
• | + mon: add a cache layer over MonitorDBStore
• + Pull request 5715
• |\
• | + rgw: url encode exposed bucket
• + Pull request 5716
• |\
• | + rgw: avoid using slashes for generated secret keys
• + Pull request 5717
• |\
• | + rgw: api adjustment following a rebase
• | + rgw: orphans, fix check on number of shards
• | + rgw: orphans, change default number of shards
• | + rgw: change error output related to orphans
• | + rgw: orphan, fix truncated detection
• | + radosgw-admin: simplify orphan command
• | + radosgw-admin: stat orphan objects before reporting leakage
• | + radosgw-admin: orphans finish command
• | + rgw: cannot re-init an orphan scan job
• | + rgw: stat_async() sets the object locator appropriately
• | + rgw: list_objects() sets namespace appropriately
• | + rgw: modify orphan search fingerprints
• | + rgw: compare oids and dump leaked objects
• | + rgw: keep accurate state for linked objects orphan scan
• | + rgw: iterate over linked objects, store them
• | + rgw: add rgw_obj::parse_raw_oid()
• | + rgw: iterate asynchronously over linked objects
• | + rgw: async object stat functionality
• | + rgw-admin: build index of bucket indexes
• | + rgw: initial work of orphan detection tool implementation
• | + Avoid an extra read on the atomic variable
• | + RGW: Make RADOS handles in RGW to be a configurable option
• + Pull request 5719
• |\
• | + rgw:segmentation fault when rgw_gc_max_objs > HASH_PRIME
• + Pull request 5720
• |\
• | + rgw:the arguments 'domain' should not be assigned when return false
• + Pull request 5721
• |\
• | + rgw: rework X-Trans-Id header to be conform with Swift API.
• | + Transaction Id added in response
• + Pull request 5732
• + rgw: init some manifest fields when handling explicit objs

./virtualenv/bin/teuthology-suite --priority 1000 --subset $(expr $RANDOM % 5)/5 --suite rgw --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports --machine-type plana,burnupi,mira

• pass http://pulpito.ceph.com/loic-2015-09-01_17:25:32-rgw-hammer-backports---basic-multi

No need to run rgw suite on this batch because all pull requests have already been tested in the previous batch. No need to run the fs suite either because there is a single pull request for fs and it has already been tested. It is included merely because it is pending approval.

$ git rev-parse ceph/hammer-backports-loic
11265088285fe45c345a8771dfc2a39918a81857

• + Pull request 5276
• + Pull request 5367
• + Pull request 5373
• + Pull request 5377
• + Pull request 5381
• + Pull request 5382
• + Pull request 5383
• + Pull request 5498
• + Pull request 5527
• + Pull request 5697
• + Pull request 5715
• |\
• | + rgw: url encode exposed bucket
• + Pull request 5716
• + Pull request 5717
• + Pull request 5719
• |\
• | + rgw:segmentation fault when rgw_gc_max_objs > HASH_PRIME
• + Pull request 5720
• |\
• | + rgw:the arguments 'domain' should not be assigned when return false
• + Pull request 5721
• + Pull request 5732
• + Pull request 5754
• + Pull request 5755
• + Pull request 5757
• + Pull request 5758
• + Pull request 5759
• + Pull request 5761
• |\
• | + OSD: break connection->session->waiting message->connection cycle
• + Pull request 5762
• + Pull request 5763
• |\
• | + osd/PGLog: dirty_to is inclusive
• + Pull request 5764
• |\
• | + common: fix code format
• | + test: add test case for insert empty ptr when buffer rebuild
• | + common: fix insert empty ptr when bufferlist rebuild
• + Pull request 5765
• |\
• | + tests: tiering agent and proxy read
• | + osd: trigger the cache agent after a promotion
• + Pull request 5766
• |\
• | + mon: fix checks on mds add_data_pool
• + Pull request 5767
• |\
• | + librbd: Add a paramter:purge_on_error in ImageCtx::invalidate_cache().
• | + librbd: Remvoe unused func ImageCtx::read_from_cache.
• | + osdc: clean up code in ObjectCacher::Object::map_write
• | + osdc: Don't pass mutex into ObjectCacher::_wait_for_write.
• | + osdc: After write try merge bh.
• | + osdc: Make last missing bh to wake up the reader.
• | + osdc: For trust_enoent is true, there is only one extent.
• | + osdc: In _readx() only no error can tidy read result.
• + Pull request 5768
• |\
• | + lockdep: allow lockdep to be dynamically enabled/disabled
• | + tests: librbd API test cannot use private md_config_t struct
• | + tests: ensure old-format RBD tests still work
• | + librados_test_stub: implement conf get/set API methods
• | + crypto: use NSS_InitContext/NSS_ShutdownContex to avoid memory leak
• | + auth: use crypto_init_mutex to protect NSS_Shutdown()
• | + auth: reinitialize NSS modules after fork()
• + Pull request 5769
• |\
• | + librbd: prevent race condition between resize requests
• + Pull request 5770
• + rgw: init some manifest fields when handling explicit objs

rbd¶

./virtualenv/bin/teuthology-suite --priority 1000 --suite rbd --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-09-02_13:56:47-rbd-hammer-backports-loic---basic-multi/
  • 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=11265088285fe45c345a8771dfc2a39918a81857 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_FEATURES=13 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd.sh'
    • rbd/librbd/{cache/none.yaml cachepool/none.yaml clusters/fixed-3.yaml copy-on-read/on.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/c_api_tests_with_object_map.yaml}
  • 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=11265088285fe45c345a8771dfc2a39918a81857 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/qemu-iotests.sh'
    • rbd/singleton/{all/qemu-iotests-writethrough.yaml}
  • 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 4'
    • rbd/librbd/{cache/writeback.yaml cachepool/small.yaml clusters/fixed-3.yaml copy-on-read/off.yaml fs/btrfs.yaml msgr-failures/few.yaml workloads/c_api_tests_with_object_map.yaml}

paddles=paddles.front.sepia.ceph.com
run=loic-2015-09-02_13:56:47-rbd-hammer-backports-loic---basic-multi
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --priority 101 --suite rbd --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• pass http://pulpito.ceph.com/loic-2015-09-06_13:56:39-rbd-hammer-backports-loic---basic-multi/

rados¶

./virtualenv/bin/teuthology-suite --priority 1000 --suite rados --subset $(expr $RANDOM % 18)/18 --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-09-02_13:58:31-rados-hammer-backports-loic---basic-multi/
  • 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=11265088285fe45c345a8771dfc2a39918a81857 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rados/test.sh'
    • rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/fastclose.yaml thrashers/morepggrow.yaml workloads/rados_api_tests.yaml}
  • saw valgrind issues
    • rados/verify/{1thrash/default.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml}

paddles=paddles.front.sepia.ceph.com
run=loic-2015-09-02_13:58:31-rados-hammer-backports-loic---basic-multi
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
./virtualenv/bin/teuthology-suite --priority 101 --suite rados --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• pass http://pulpito.ceph.com/loic-2015-09-06_13:52:47-rados-hammer-backports-loic---basic-multi/

Verifying that https://github.com/ceph/ceph/pull/5887 does not cause problems with the rados suite.

teuthology-suite --priority 101 --suite rados --subset $(expr $RANDOM % 18)/18 --suite-branch hammer --distro ubuntu --email loic@dachary.org --filter-out=rhel --ceph wip-pr-5887 --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-10-03_11:11:28-rados-wip-pr-5887---basic-multi/
  • bug more hammer git ceph com updates
    • rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/fastclose.yaml thrashers/mapgap.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/osd-delay.yaml thrashers/default.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/fastclose.yaml thrashers/morepggrow.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/osd-delay.yaml thrashers/mapgap.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/fastclose.yaml thrashers/pggrow.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/osd-delay.yaml thrashers/morepggrow.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/fastclose.yaml thrashers/default.yaml workloads/admin_socket_objecter_requests.yaml}
    • rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/admin_socket_objecter_requests.yaml}
  • 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 5'
    • rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/osd-delay.yaml thrashers/default.yaml workloads/cache-agent-big.yaml}
  • qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45
    • rados/singleton/{all/cephtool.yaml fs/btrfs.yaml msgr-failures/few.yaml}
  • "2015-10-04 05:34:47.220275 osd.1 10.214.136.12:6804/62701 10 : cluster [ERR] 9.0s0 shard 2(1): soid 3df68405/repair_test_obj/head//9 extra attr hinfo_key" in cluster log
    • rados/basic/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/many.yaml tasks/repair_test.yaml}

run=loic-2015-10-03_11:11:28-rados-wip-pr-5887---basic-multi
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
teuthology-suite --priority 101 --suite rados --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph wip-pr-5887 --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-10-04_15:14:57-rados-wip-pr-5887---basic-multi
  • already fixed qa/workunits/cephtool/test.sh: don't assume crash_replay_interval=45
    • rados/singleton/{all/cephtool.yaml fs/btrfs.yaml msgr-failures/few.yaml}

powercycle¶

./virtualenv/bin/teuthology-suite -l2 -v -c hammer-backports-loic -k testing -m plana,burnupi,mira -s powercycle -p 1000 --email loic@dachary.org

• pass http://pulpito.ceph.com/loic-2015-09-02_14:06:51-powercycle-hammer-backports-loic-testing-basic-multi/

upgrade¶

teuthology-suite --verbose --suite upgrade/hammer --filter=centos_6,ubuntu_14 --suite-branch hammer --ceph hammer-backports --machine-type vps --priority 1000

• fail http://pulpito.ceph.com/loic-2015-09-02_14:09:47-upgrade:hammer-hammer-backports-loic---basic-vps
  • timeout waiting for machines **
    • upgrade:hammer/older/{0-cluster/start.yaml 1-install/v0.94.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrados.yaml} distros/centos_6.5.yaml}
    • upgrade:hammer/older/{0-cluster/start.yaml 1-install/v0.94.1.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrados.yaml} distros/centos_6.5.yaml}
  • "sudo find /var/log/ceph name '.log' -print0 | sudo xargs -0 --no-run-if-empty - gzip --"*
    • upgrade:hammer/newer/{0-cluster/start.yaml 1-install/v0.94.2.yaml 2-workload/rbd.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/centos_6.5.yaml}

ceph-deploy¶

teuthology-suite --verbose --suite ceph-deploy --filter=centos_6,ubuntu_14 --suite-branch hammer --ceph hammer-backports-loic --machine-type vps --priority 1000

• fail http://pulpito.ceph.com/loic-2015-09-02_14:58:36-ceph-deploy-hammer-backports-loic---basic-vps
  • ceph health was unable to get 'HEALTH_OK' after waiting 15 minutes
    • ceph-deploy/basic/{ceph-deploy-overrides/enable_dmcrypt_diff_journal_disk.yaml config_options/cephdeploy_conf.yaml distros/centos_6.5.yaml tasks/ceph-deploy_hello_world.yaml}
    • ceph-deploy/basic/{ceph-deploy-overrides/ceph_deploy_dmcrypt.yaml config_options/cephdeploy_conf.yaml distros/centos_6.5.yaml tasks/ceph-deploy_hello_world.yaml}

Note: centos 6.5 was removed from distros

fs¶

./virtualenv/bin/teuthology-suite --priority 1000 --suite fs --subset $(expr $RANDOM % 5)/5 --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-09-06_23:31:17-fs-hammer-backports-loic---basic-multi/
  • 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=11265088285fe45c345a8771dfc2a39918a81857 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs/test.sh'
    • fs/basic/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml}
    • fs/verify/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}

run=loic-2015-09-06_23:31:17-fs-hammer-backports-loic---basic-multi
paddles=paddles.front.sepia.ceph.com
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
teuthology-suite --priority 1000 --suite fs --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-09-08_13:46:08-fs-hammer-backports-loic---basic-multi/
  • , 'failed': True, 'item': 'http://git.ceph.com/release.asc', 'msg': 'Failed to download key at http://git.ceph.com/release.asc: Request failed: <urlopen error timed out>'}, 'burnupi19.front.sepia.ceph.com': {'invocation': {'module_name': 'apt_key', 'module_args': ''}, 'failed': True, 'item': 'http://git.ceph.com/release.asc', 'msg': 'Failed to download key at http://git.ceph.com/release.asc: Request failed: <urlopen error timed out>'}, 'plana65.front.sepia.ceph.com': {'invocation': {'module_name': 'apt_key', 'module_args': ''}, 'failed': True, 'item': 'http://git.ceph.com/release.asc', 'msg': 'Failed to download key at http://git.ceph.com/release.asc: Request failed: <urlopen error timed out>'}}
    • fs/verify/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml validater/lockdep.yaml}
  • 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=11265088285fe45c345a8771dfc2a39918a81857 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/libcephfs/test.sh'
    • fs/basic/{clusters/fixed-3-cephfs.yaml debug/mds_client.yaml fs/btrfs.yaml inline/yes.yaml overrides/whitelist_wrongly_marked_down.yaml tasks/libcephfs_interface_tests.yaml} known bug http://tracker.ceph.com/issues/12598

Assuming transient errors / environmental errors, running again

run=loic-2015-09-06_23:31:17-fs-hammer-backports-loic---basic-multi
paddles=paddles.front.sepia.ceph.com
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
teuthology-suite --priority 1000 --suite fs --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph hammer-backports-loic --machine-type plana,burnupi,mira

• fail http://pulpito.ceph.com/loic-2015-09-09_14:13:23-fs-hammer-backports-loic---basic-multi/

The errors are LibCephFS.GetPoolId failure, verifying the https://github.com/ceph/ceph/pull/5887

run=loic-2015-09-06_23:31:17-fs-hammer-backports-loic---basic-multi
paddles=paddles.front.sepia.ceph.com
eval filter=$(curl --silent http://$paddles/runs/$run/ | jq '.jobs[] | select(.status == "dead" or .status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
teuthology-suite --priority 101 --suite fs --filter="$filter" --suite-branch hammer --distro ubuntu --email loic@dachary.org --ceph wip-pr-5887 --machine-type plana,burnupi,mira

• pass http://pulpito.ceph.com/loic-2015-10-03_10:10:17-fs-wip-pr-5887---basic-multi/

QE Validation (started 10/12/15)¶

re-runs command lines and filters are captured in http://pad.ceph.com/p/hammer_v0.94.4_QE_validation_notes

==