Bug #11359: cephtool/test.sh:test_tiering intermittent failure - Ceph - Ceph

Actions

#1

Updated by Loïc Dachary about 9 years ago

Status changed from New to 12
Assignee set to Loïc Dachary

loic@alien2:/data/loic/ceph/src$ ./ceph -c testdir/test-7202/ceph.conf -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
    cluster 4f654df0-eef4-4c44-bacd-029211fb3d27
     health HEALTH_OK
     monmap e1: 3 mons at {a=127.0.0.1:7202/0,b=127.0.0.1:7203/0,c=127.0.0.1:7204/0}
            election epoch 6, quorum 0,1,2 a,b,c
     osdmap e84: 3 osds: 3 up, 3 in
      pgmap v187: 12 pgs, 3 pools, 20480 bytes data, 5 objects
            135 GB used, 1325 GB / 1476 GB avail
                  12 active+clean
loic@alien2:/data/loic/ceph/src$ ./ceph -c testdir/test-7202/ceph.conf df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
GLOBAL:
    SIZE      AVAIL     RAW USED     %RAW USED
    1476G     1325G         135G          9.20
POOLS:
    NAME         ID     USED      %USED     MAX AVAIL     OBJECTS
    rbd          0          0         0          441G           0
    datapool     10     16384         0          441G           4
    cache4       11      4096         0          441G           1
loic@alien2:/data/loic/ceph/src$ ./rados -c testdir/test-7202/ceph.conf ls -p cache4
foo5
loic@alien2:/data/loic/ceph/src$ uname -a
Linux alien2 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
loic@alien2:/data/loic/ceph/src$

Actions

Copy link

#2

Updated by Loïc Dachary about 9 years ago

test_tiering: 372: for f in '`seq 1 5`'
 test_tiering: 373: rados -p cache4 put foo1 /tmp/tmp.LzstMXOq39
 test_tiering: 372: for f in '`seq 1 5`'
 test_tiering: 373: rados -p cache4 put foo2 /tmp/tmp.LzstMXOq39
 test_tiering: 372: for f in '`seq 1 5`'
 test_tiering: 373: rados -p cache4 put foo3 /tmp/tmp.LzstMXOq39
 test_tiering: 372: for f in '`seq 1 5`'
 test_tiering: 373: rados -p cache4 put foo4 /tmp/tmp.LzstMXOq39
 test_tiering: 372: for f in '`seq 1 5`'
 test_tiering: 373: rados -p cache4 put foo5 /tmp/tmp.LzstMXOq39
 test_tiering: 375: rm -f /tmp/tmp.LzstMXOq39
 test_tiering: 376: ceph df
 test_tiering: 376: grep ' 5 '
 test_tiering: 376: grep cache4
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_tiering: 377: echo waiting for pg stats to flush
waiting for pg stats to flush
 test_tiering: 378: sleep 2
 test_tiering: 376: grep ' 5 '
 test_tiering: 376: grep cache4
 test_tiering: 376: ceph df
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_tiering: 377: echo waiting for pg stats to flush
waiting for pg stats to flush
 test_tiering: 378: sleep 2
 test_tiering: 376: ceph df
 test_tiering: 376: grep cache4
 test_tiering: 376: grep ' 5 '
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
    cache4       11     20480         0        28838M           5 
 test_tiering: 380: ceph health
 test_tiering: 380: grep WARN
 test_tiering: 380: grep cache4
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_tiering: 1: rm -fr /tmp/cephtool5605
+ test 1 = 124
+ success=false
+ false

Actions

Copy link

#3

Updated by Loïc Dachary about 9 years ago

Priority changed from Normal to Urgent

upgrading to urgent because we need the tests that were disabled. It fixed the intermittent failure by removing the tests which is a bad solution. My fault.

Actions

Copy link

#4

Updated by Loïc Dachary almost 9 years ago

Status changed from 12 to In Progress
Regression set to No

disabled the tests temporarily https://github.com/ceph/ceph/pull/4337

Actions

Copy link

#5

Updated by Loïc Dachary almost 9 years ago

master https://github.com/ceph/ceph/pull/4662

Actions

Copy link

#6

Updated by Loïc Dachary almost 9 years ago

Status changed from In Progress to Fix Under Review

Actions

Copy link

#7

Updated by Loïc Dachary almost 9 years ago

Status changed from Fix Under Review to Resolved

Actions

Copy link

#8

Updated by Loïc Dachary almost 9 years ago

Status changed from Resolved to 12

*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' is now (or already was) a cache tier of 'datapool'
 test_tiering: 336: ceph osd tier cache-mode cachepool writeback
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set cache-mode for pool 'cachepool' to writeback
 test_tiering: 337: rados -p datapool put object /etc/passwd
 test_tiering: 338: rados -p cachepool stat object
cachepool/object mtime 2015-05-19 08:36:10.000000, size 656
 test_tiering: 339: rados -p cachepool cache-flush object
 test_tiering: 340: rados -p datapool stat object
datapool/object mtime 2015-05-19 08:36:10.000000, size 656
 test_tiering: 341: ceph osd tier remove-overlay datapool
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
there is now (or already was) no overlay for 'datapool'
 test_tiering: 342: ceph osd tier remove datapool cachepool
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' is now (or already was) not a tier of 'datapool'
 test_tiering: 343: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' removed
 test_tiering: 344: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'datapool' removed
 test_tiering: 347: ceph osd pool create datapool 2
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'datapool' created
 test_tiering: 348: ceph osd pool create cachepool 2
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' created
 test_tiering: 349: ceph osd tier add-cache datapool cachepool 1024000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' is now (or already was) a cache tier of 'datapool'
 test_tiering: 350: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it
 test_tiering: 350: true
 test_tiering: 351: check_response 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\'''
 check_response: 108: expected_string='EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\'''
 check_response: 109: retcode=
 check_response: 110: expected_retcode=
 check_response: 111: '[' '' -a '!=' ']'
 check_response: 116: grep --quiet -- 'EBUSY: pool '\''cachepool'\'' is a tier of '\''datapool'\''' /tmp/cephtool5232/test_invalid.5232
 test_tiering: 352: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it
 test_tiering: 352: true
 test_tiering: 353: check_response 'EBUSY: pool '\''datapool'\'' has tiers cachepool'
 check_response: 108: expected_string='EBUSY: pool '\''datapool'\'' has tiers cachepool'
 check_response: 109: retcode=
 check_response: 110: expected_retcode=
 check_response: 111: '[' '' -a '!=' ']'
 check_response: 116: grep --quiet -- 'EBUSY: pool '\''datapool'\'' has tiers cachepool' /tmp/cephtool5232/test_invalid.5232
 test_tiering: 354: ceph osd tier remove-overlay datapool
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
there is now (or already was) no overlay for 'datapool'
 test_tiering: 355: ceph osd tier remove datapool cachepool
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' is now (or already was) not a tier of 'datapool'
 test_tiering: 356: ceph osd pool delete cachepool cachepool --yes-i-really-really-mean-it
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cachepool' removed
 test_tiering: 357: ceph osd pool delete datapool datapool --yes-i-really-really-mean-it
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'datapool' removed
 test_tiering: 360: ceph osd set notieragent
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set notieragent
 test_tiering: 361: ceph osd pool create datapool 2
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'datapool' created
 test_tiering: 362: ceph osd pool create cache4 2
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cache4' created
 test_tiering: 363: ceph osd tier add-cache datapool cache4 1024000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cache4' is now (or already was) a cache tier of 'datapool'
 test_tiering: 364: ceph osd tier cache-mode cache4 writeback
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set cache-mode for pool 'cache4' to writeback
  test_tiering: 365: mktemp
  test_tiering: 365: grep tmp
 test_tiering: 365: tmpfile=/tmp/tmp.e5Zcu7Fjjb
 test_tiering: 366: dd if=/dev/zero of=/tmp/tmp.e5Zcu7Fjjb bs=4K count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000108984 s, 37.6 MB/s
 test_tiering: 367: ceph osd pool set cache4 target_max_objects 200
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set pool 11 target_max_objects to 200
 test_tiering: 368: ceph osd pool set cache4 target_max_bytes 1000000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set pool 11 target_max_bytes to 1000000
 test_tiering: 369: rados -p cache4 put foo1 /tmp/tmp.e5Zcu7Fjjb
 test_tiering: 370: rados -p cache4 put foo2 /tmp/tmp.e5Zcu7Fjjb
 test_tiering: 371: rm -f /tmp/tmp.e5Zcu7Fjjb
 test_tiering: 372: ceph tell 'osd.*' flush_pg_stats
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_tiering: 373: ceph df
 test_tiering: 373: grep cache4
 test_tiering: 373: grep ' 2 '
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_tiering: 1: rm -fr /tmp/cephtool5232
+ test 1 = 124
+ success=false

http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/5322/console

Actions

Copy link

#9

Updated by Loïc Dachary almost 9 years ago

Status changed from 12 to Resolved

did not show up again, let's assume it was a run that had a version of the tree without the patch resolving this issue

Project

General

Profile

Ceph

Custom queries

Bug #11359

cephtool/test.sh:test_tiering intermittent failure

Updated by Loïc Dachary about 9 years ago

Updated by Loïc Dachary about 9 years ago

Updated by Loïc Dachary about 9 years ago

Updated by Loïc Dachary almost 9 years ago

Updated by Loïc Dachary almost 9 years ago

Updated by Loïc Dachary almost 9 years ago

Updated by Loïc Dachary almost 9 years ago

Updated by Loïc Dachary almost 9 years ago

Updated by Loïc Dachary almost 9 years ago