Actions
Bug #11359
closedcephtool/test.sh:test_tiering intermittent failure
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Sometime the following happens:
test_tiering: 372: seq 1 5 test_tiering: 372: for f in '`seq 1 5`' test_tiering: 373: rados -p cache4 put foo1 /tmp/tmp.0KVbmre2Qx test_tiering: 372: for f in '`seq 1 5`' test_tiering: 373: rados -p cache4 put foo2 /tmp/tmp.0KVbmre2Qx test_tiering: 372: for f in '`seq 1 5`' test_tiering: 373: rados -p cache4 put foo3 /tmp/tmp.0KVbmre2Qx test_tiering: 372: for f in '`seq 1 5`' test_tiering: 373: rados -p cache4 put foo4 /tmp/tmp.0KVbmre2Qx test_tiering: 372: for f in '`seq 1 5`' test_tiering: 373: rados -p cache4 put foo5 /tmp/tmp.0KVbmre2Qx test_tiering: 375: rm -f /tmp/tmp.0KVbmre2Qx test_tiering: 376: ceph df test_tiering: 376: grep cache4 test_tiering: 376: grep ' 5 ' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_tiering: 377: echo waiting for pg stats to flush waiting for pg stats to flush test_tiering: 378: sleep 2 test_tiering: 376: ceph df test_tiering: 376: grep cache4 test_tiering: 376: grep ' 5 ' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_tiering: 377: echo waiting for pg stats to flush waiting for pg stats to flush test_tiering: 378: sleep 2 test_tiering: 376: ceph df test_tiering: 376: grep cache4 test_tiering: 376: grep ' 5 ' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_tiering: 377: echo waiting for pg stats to flush
and continues forever. The cluster is at
loic@ceph-ubuntu-14-04-i386:~/ceph/src$ ./ceph -c testdir/test-7202/ceph.conf df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
47993M 6638M 38849M 80.95
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 2212M 0
datapool 10 16384 0 2212M 4
cache4 11 4096 0 2212M 1
loic@ceph-ubuntu-14-04-i386:~/ceph/src$ ./rados -c testdir/test-7202/ceph.conf -p cache4 ls
foo5
and won't change.
Updated by Loïc Dachary about 9 years ago
- Status changed from New to 12
- Assignee set to Loïc Dachary
loic@alien2:/data/loic/ceph/src$ ./ceph -c testdir/test-7202/ceph.conf -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
cluster 4f654df0-eef4-4c44-bacd-029211fb3d27
health HEALTH_OK
monmap e1: 3 mons at {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
election epoch 6, quorum 0,1,2 a,b,c
osdmap e84: 3 osds: 3 up, 3 in
pgmap v187: 12 pgs, 3 pools, 20480 bytes data, 5 objects
135 GB used, 1325 GB / 1476 GB avail
12 active+clean
loic@alien2:/data/loic/ceph/src$ ./ceph -c testdir/test-7202/ceph.conf df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
1476G 1325G 135G 9.20
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 441G 0
datapool 10 16384 0 441G 4
cache4 11 4096 0 441G 1
loic@alien2:/data/loic/ceph/src$ ./rados -c testdir/test-7202/ceph.conf ls -p cache4
foo5
loic@alien2:/data/loic/ceph/src$ uname -a
Linux alien2 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
loic@alien2:/data/loic/ceph/src$
Updated by Loïc Dachary about 9 years ago
test_tiering: 372: for f in '`seq 1 5`'
test_tiering: 373: rados -p cache4 put foo1 /tmp/tmp.LzstMXOq39
test_tiering: 372: for f in '`seq 1 5`'
test_tiering: 373: rados -p cache4 put foo2 /tmp/tmp.LzstMXOq39
test_tiering: 372: for f in '`seq 1 5`'
test_tiering: 373: rados -p cache4 put foo3 /tmp/tmp.LzstMXOq39
test_tiering: 372: for f in '`seq 1 5`'
test_tiering: 373: rados -p cache4 put foo4 /tmp/tmp.LzstMXOq39
test_tiering: 372: for f in '`seq 1 5`'
test_tiering: 373: rados -p cache4 put foo5 /tmp/tmp.LzstMXOq39
test_tiering: 375: rm -f /tmp/tmp.LzstMXOq39
test_tiering: 376: ceph df
test_tiering: 376: grep ' 5 '
test_tiering: 376: grep cache4
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_tiering: 377: echo waiting for pg stats to flush
waiting for pg stats to flush
test_tiering: 378: sleep 2
test_tiering: 376: grep ' 5 '
test_tiering: 376: grep cache4
test_tiering: 376: ceph df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_tiering: 377: echo waiting for pg stats to flush
waiting for pg stats to flush
test_tiering: 378: sleep 2
test_tiering: 376: ceph df
test_tiering: 376: grep cache4
test_tiering: 376: grep ' 5 '
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
cache4 11 20480 0 28838M 5
test_tiering: 380: ceph health
test_tiering: 380: grep WARN
test_tiering: 380: grep cache4
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_tiering: 1: rm -fr /tmp/cephtool5605
+ test 1 = 124
+ success=false
+ false
Updated by Loïc Dachary almost 9 years ago
- Priority changed from Normal to Urgent
upgrading to urgent because we need the tests that were disabled. It fixed the intermittent failure by removing the tests which is a bad solution. My fault.
Updated by Loïc Dachary almost 9 years ago
- Status changed from 12 to In Progress
- Regression set to No
- disabled the tests temporarily https://github.com/ceph/ceph/pull/4337
Updated by Loïc Dachary almost 9 years ago
Updated by Loïc Dachary almost 9 years ago
- Status changed from In Progress to Fix Under Review
Updated by Loïc Dachary almost 9 years ago
- Status changed from Fix Under Review to Resolved
Updated by Loïc Dachary almost 9 years ago
- Status changed from Resolved to 12
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' is now (or already was) a cache tier of 'datapool' test_tiering: 336: ceph osd tier cache-mode cachepool writeback *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set cache-mode for pool 'cachepool' to writeback test_tiering: 337: rados -p datapool put object /etc/passwd test_tiering: 338: rados -p cachepool stat object cachepool/object mtime 2015-05-19 08:36:10.000000, size 656 test_tiering: 339: rados -p cachepool cache-flush object test_tiering: 340: rados -p datapool stat object datapool/object mtime 2015-05-19 08:36:10.000000, size 656 test_tiering: 341: ceph osd tier remove-overlay datapool *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** there is now (or already was) no overlay for 'datapool' test_tiering: 342: ceph osd tier remove datapool cachepool *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' is now (or already was) not a tier of 'datapool' test_tiering: 343: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' removed test_tiering: 344: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'datapool' removed test_tiering: 347: ceph osd pool create datapool 2 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'datapool' created test_tiering: 348: ceph osd pool create cachepool 2 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' created test_tiering: 349: ceph osd tier add-cache datapool cachepool 1024000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' is now (or already was) a cache tier of 'datapool' test_tiering: 350: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it test_tiering: 350: true test_tiering: 351: check_response 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' check_response: 108: expected_string='EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' check_response: 109: retcode= check_response: 110: expected_retcode= check_response: 111: '[' '' -a '!=' ']' check_response: 116: grep --quiet -- 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' /tmp/cephtool5232/test_invalid.5232 test_tiering: 352: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it test_tiering: 352: true test_tiering: 353: check_response 'EBUSY: pool '\''datapool'\'' has tiers cachepool' check_response: 108: expected_string='EBUSY: pool '\''datapool'\'' has tiers cachepool' check_response: 109: retcode= check_response: 110: expected_retcode= check_response: 111: '[' '' -a '!=' ']' check_response: 116: grep --quiet -- 'EBUSY: pool '\''datapool'\'' has tiers cachepool' /tmp/cephtool5232/test_invalid.5232 test_tiering: 354: ceph osd tier remove-overlay datapool *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** there is now (or already was) no overlay for 'datapool' test_tiering: 355: ceph osd tier remove datapool cachepool *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' is now (or already was) not a tier of 'datapool' test_tiering: 356: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cachepool' removed test_tiering: 357: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'datapool' removed test_tiering: 360: ceph osd set notieragent *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set notieragent test_tiering: 361: ceph osd pool create datapool 2 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'datapool' created test_tiering: 362: ceph osd pool create cache4 2 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cache4' created test_tiering: 363: ceph osd tier add-cache datapool cache4 1024000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cache4' is now (or already was) a cache tier of 'datapool' test_tiering: 364: ceph osd tier cache-mode cache4 writeback *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set cache-mode for pool 'cache4' to writeback test_tiering: 365: mktemp test_tiering: 365: grep tmp test_tiering: 365: tmpfile=/tmp/tmp.e5Zcu7Fjjb test_tiering: 366: dd if=/dev/zero of=/tmp/tmp.e5Zcu7Fjjb bs=4K count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000108984 s, 37.6 MB/s test_tiering: 367: ceph osd pool set cache4 target_max_objects 200 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set pool 11 target_max_objects to 200 test_tiering: 368: ceph osd pool set cache4 target_max_bytes 1000000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set pool 11 target_max_bytes to 1000000 test_tiering: 369: rados -p cache4 put foo1 /tmp/tmp.e5Zcu7Fjjb test_tiering: 370: rados -p cache4 put foo2 /tmp/tmp.e5Zcu7Fjjb test_tiering: 371: rm -f /tmp/tmp.e5Zcu7Fjjb test_tiering: 372: ceph tell 'osd.*' flush_pg_stats *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_tiering: 373: ceph df test_tiering: 373: grep cache4 test_tiering: 373: grep ' 2 ' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_tiering: 1: rm -fr /tmp/cephtool5232 + test 1 = 124 + success=false
http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/5322/console
Updated by Loïc Dachary almost 9 years ago
- Status changed from 12 to Resolved
did not show up again, let's assume it was a run that had a version of the tree without the patch resolving this issue
Actions