Project

General

Profile

Actions
Copy link

Bug #45119

closed

problems rerunning failed jobs

Added by Yuri Weinstein about 4 years ago. Updated about 4 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
-
Category:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

consider this use case:

Run http://pulpito.ceph.com/yuriw-2020-04-15_00:09:40-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi/

Build - https://shaman.ceph.com/builds/ceph/wip-yuri-testing-2020-04-14-1606-nautilus/183619590533bff9e6201956d2f56d46831725eb/

Command lines:

CEPH_BRANCH=wip-yuri-testing-2020-04-14-1606-nautilus
MACHINE_NAME=smithi
CEPH_REPO=https://github.com/ceph/ceph-ci.git

RERUN=yuriw-2020-04-15_00:09:40-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi

teuthology-suite -v -m $MACHINE_NAME -c $CEPH_BRANCH --rerun $RERUN -p 91 -R fail,dead,running,waiting --dry-run

teuthology-suite -v -m $MACHINE_NAME -c $CEPH_BRANCH --rerun $RERUN -p 91 -R fail,dead,running,waiting --dry-run
2020-04-16 19:35:48,436.436 INFO:teuthology.suite:Using stored seed=4466
2020-04-16 19:35:48,436.436 INFO:teuthology.suite:Using stored subset=(1, 10)
2020-04-16 19:35:48,437.437 INFO:teuthology.suite.run:kernel sha1: distro
2020-04-16 19:35:48,687.687 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci wip-yuri-testing-2020-04-14-1606-nautilus -> 183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:48,687.687 INFO:teuthology.suite.run:ceph sha1: 183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:48,687.687 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2020-04-16 19:35:48,688.688 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:48,919.919 INFO:teuthology.suite.run:ceph version: 14.2.8-16.g1836195
2020-04-16 19:35:49,147.147 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/teuthology master -> e642842840a163f9a27c58bdc605fb44bf4f6b1c
2020-04-16 19:35:49,148.148 INFO:teuthology.suite.run:teuthology branch: master e642842840a163f9a27c58bdc605fb44bf4f6b1c
2020-04-16 19:35:49,195.195 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci.git wip-yuri-testing-2020-04-14-1606-nautilus -> 183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:49,238.238 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci.git wip-yuri-testing-2020-04-14-1606-nautilus -> 183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:49,239.239 INFO:teuthology.suite.run:ceph-ci branch: wip-yuri-testing-2020-04-14-1606-nautilus 183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:49,241.241 INFO:teuthology.repo_utils:/home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus was just updated; assuming it is current
2020-04-16 19:35:49,241.241 INFO:teuthology.repo_utils:Resetting repo at /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus to branch origin/wip-yuri-testing-2020-04-14-1606-nautilus
2020-04-16 19:35:49,322.322 DEBUG:teuthology.suite.run:Suite kcephfs in /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs
2020-04-16 19:35:49,323.323 INFO:teuthology.suite.build_matrix:Subset=1/10
2020-04-16 19:35:49,349.349 INFO:teuthology.suite.run:Suite kcephfs in /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs generated 70 jobs (not yet filtered)
2020-04-16 19:35:49,353.353 DEBUG:teuthology.suite.run:Base job config:
branch: wip-yuri-testing-2020-04-14-1606-nautilus
kernel:
  kdb: true
  sha1: distro
machine_type: smithi
name: yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi
nuke-on-error: true
overrides:
  admin_socket:
    branch: wip-yuri-testing-2020-04-14-1606-nautilus
  ceph:
    conf:
      mgr:
        debug mgr: 20
        debug ms: 1
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 20
        debug osd: 25
    log-whitelist:
    - \(MDS_ALL_DOWN\)
    - \(MDS_UP_LESS_THAN_MAX\)
    sha1: 183619590533bff9e6201956d2f56d46831725eb
  ceph-deploy:
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        osd default pool size: 2
  install:
    ceph:
      sha1: 183619590533bff9e6201956d2f56d46831725eb
  workunit:
    branch: wip-yuri-testing-2020-04-14-1606-nautilus
    sha1: 183619590533bff9e6201956d2f56d46831725eb
priority: 91
repo: git://git.ceph.com/ceph-ci.git
sha1: 183619590533bff9e6201956d2f56d46831725eb
sleep_before_teardown: 0
suite: kcephfs
suite_branch: wip-yuri-testing-2020-04-14-1606-nautilus
suite_relpath: qa
suite_repo: git://git.ceph.com/ceph-ci.git
suite_sha1: 183619590533bff9e6201956d2f56d46831725eb
tasks: []
teuthology_branch: master
2020-04-16 19:35:49,375.375 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2020-04-16 19:35:49,376.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:49,644.644 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2020-04-16 19:35:49,644.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=183619590533bff9e6201956d2f56d46831725eb
2020-04-16 19:35:49,937.937 DEBUG:teuthology.suite.run:Base job config:
branch: wip-yuri-testing-2020-04-14-1606-nautilus
kernel:
  kdb: true
  sha1: distro
machine_type: smithi
name: yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi
nuke-on-error: true
overrides:
  admin_socket:
    branch: wip-yuri-testing-2020-04-14-1606-nautilus
  ceph:
    conf:
      mgr:
        debug mgr: 20
        debug ms: 1
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 20
        debug osd: 25
    log-whitelist:
    - \(MDS_ALL_DOWN\)
    - \(MDS_UP_LESS_THAN_MAX\)
    sha1: 183619590533bff9e6201956d2f56d46831725eb
  ceph-deploy:
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        osd default pool size: 2
  install:
    ceph:
      sha1: 183619590533bff9e6201956d2f56d46831725eb
  workunit:
    branch: wip-yuri-testing-2020-04-14-1606-nautilus
    sha1: 183619590533bff9e6201956d2f56d46831725eb
priority: 91
repo: git://git.ceph.com/ceph-ci.git
sha1: 183619590533bff9e6201956d2f56d46831725eb
sleep_before_teardown: 0
suite: kcephfs
suite_branch: wip-yuri-testing-2020-04-14-1606-nautilus
suite_relpath: qa
suite_repo: git://git.ceph.com/ceph-ci.git
suite_sha1: 183619590533bff9e6201956d2f56d46831725eb
tasks: []
teuthology_branch: master
2020-04-16 19:35:49,940.940 INFO:teuthology.suite.util:Memo: /home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --first-in-suite --subset 1/10 --seed 4466
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.run:Scheduling kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-recovery.yaml}
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.util:/home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --description 'kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/client-recovery.yaml}' -- /tmp/schedule_suite_h_8kkrze /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/begin.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/clusters/1-mds-4-client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mds.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mon.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/osd.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/mount.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/k-distro.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/rhel_latest.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/ms-die-on-skipped.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/objectstore-ec/bluestore-comp-ec-root.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/frag_enable.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/log-config.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/osd-asserts.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_health.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_wrongly_marked_down.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/tasks/client-recovery.yaml
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.run:Scheduling kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/damage.yaml}
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.util:/home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --description 'kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/random/{k-testing.yaml supported$/{rhel_latest.yaml}} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/damage.yaml}' -- /tmp/schedule_suite_h_8kkrze /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/begin.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/clusters/1-mds-4-client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mds.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mon.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/osd.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/mount.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/random/k-testing.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/random/supported$/rhel_latest.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/ms-die-on-skipped.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/objectstore-ec/bluestore-comp.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/frag_enable.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/log-config.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/osd-asserts.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_health.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_wrongly_marked_down.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/tasks/damage.yaml
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.run:Scheduling kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/data-scan.yaml}
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.util:/home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --description 'kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-ec-root.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/data-scan.yaml}' -- /tmp/schedule_suite_h_8kkrze /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/begin.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/clusters/1-mds-4-client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mds.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mon.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/osd.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/mount.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/k-distro.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/rhel_latest.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/ms-die-on-skipped.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/objectstore-ec/bluestore-ec-root.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/frag_enable.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/log-config.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/osd-asserts.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_health.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_wrongly_marked_down.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/tasks/data-scan.yaml
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.run:Scheduling kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/forward-scrub.yaml}
2020-04-16 19:35:49,941.941 INFO:teuthology.suite.util:/home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --description 'kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_latest.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/forward-scrub.yaml}' -- /tmp/schedule_suite_h_8kkrze /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/begin.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/clusters/1-mds-4-client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/client.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mds.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/mon.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/conf/osd.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/mount.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/k-distro.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/distro/rhel/rhel_latest.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/kclient/overrides/ms-die-on-skipped.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/objectstore-ec/bluestore-bitmap.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/frag_enable.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/log-config.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/osd-asserts.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_health.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/overrides/whitelist_wrongly_marked_down.yaml /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs/recovery/tasks/forward-scrub.yaml
2020-04-16 19:35:49,942.942 INFO:teuthology.suite.run:Suite kcephfs in /home/yuriw/src/git.ceph.com_ceph-c_wip-yuri-testing-2020-04-14-1606-nautilus/qa/suites/kcephfs scheduled 4 jobs.
2020-04-16 19:35:49,942.942 INFO:teuthology.suite.run:66/70 jobs were filtered out.
2020-04-16 19:35:49,942.942 INFO:teuthology.suite.util:Results: /home/yuriw/teuthology/virtualenv/bin/teuthology-schedule --name yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi --num 1 --worker smithi --dry-run --priority 91 -v --last-in-suite --timeout 43200
2020-04-16 19:35:49,942.942 INFO:teuthology.suite.run:Test results viewable at http://pulpito.front.sepia.ceph.com:80/yuriw-2020-04-16_19:35:48-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi/

expected 6 jobs, not 4

notice "stored subset=(1, 10)" - where is it coming from ?

Related issues 1 (0 open — 1 closed)

Has duplicate teuthology - Bug #35951: Recently merged "$" feature broke --filter and --rerunDuplicate

Actions
Actions
Copy link
 #1

Updated by Kyrylo Shatskyy about 4 years ago

I guess subset things comes from results.log
Yuri can you attach it to the bug, I hope you haven't removed it. Or just grep with 'subset:'.
Needs to figure out at which level the bug is introduced.

Actions
Copy link
 #2

Updated by Kyrylo Shatskyy about 4 years ago

Yuri, okay, I've got where it from:
http://qa-proxy.ceph.com/teuthology/yuriw-2020-04-15_00:09:40-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi/results.log

2020-04-15T01:58:13.192 INFO:root:teuthology version: 1.0.0-2a52f60
2020-04-15T01:58:13.193 INFO:teuthology.results:subset: '1/10'
2020-04-15T01:58:13.193 INFO:teuthology.results:seed: '4466'

Interesting, the later in the log it has:

info:   http://pulpito.ceph.com/yuriw-2020-04-15_00:09:40-kcephfs-wip-yuri-testing-2020-04-14-1606-nautilus-distro-basic-smithi/4954392/
2020-04-15T14:10:40.774 INFO:teuthology.results:starting coverage generation
2020-04-15T14:10:40.784 ERROR:teuthology.results:error generating memo/results
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 35, in main
    int(args['--timeout']), args['--dry-run'])
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 84, in results
    generate_coverage(archive_dir, name)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/results.py", line 107, in generate_coverage
    archive_dir,
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Not sure it is related to the issue though

Actions
Copy link
 #3

Updated by Kyrylo Shatskyy about 4 years ago

Yuri, are you sure this is a new issue?

I think it is an old bug related to '$' magic, since it is generating randomized list of tests each time.
Seems like related to the https://tracker.ceph.com/issues/35951, maybe it is a duplicate.

Actions
Copy link
 #4

Updated by Yuri Weinstein about 4 years ago

It could be been masked and seems to be related to using `subset`

Main point it is not right and not expected behavior - I can be convinced otherwise :)

You want to be able to rerun failed jobs in the run and is there were 21 jobs, you expect to be able to rerun all 21.

I see where it's coming from results.log file, but I think reruns should ignore subset setting and be able to run only failed jobs.

Let's see what @Kefu Chai thinks, he knows code this better then anybody else

Actions
Copy link
 #6

Updated by Kefu Chai about 4 years ago

i updated #35951 with my findings. but it's still a mystery to me. no clues so far.

Actions
Copy link
 #7

Updated by Josh Durgin about 4 years ago

Bisected this - it was working before the py3 conversion:

16ccba3ee4b6be7f17e9c38eeaa7e4eb5640ffe7 is the first bad commit
commit 16ccba3ee4b6be7f17e9c38eeaa7e4eb5640ffe7
Author: Kyr Shatskyy <>
Date: Wed Dec 11 11:14:48 2019 +0100

bootstrap: use py3 by default

Trying to narrow down what the difference is.

Actions
Copy link
 #8

Updated by Josh Durgin about 4 years ago

The root cause is the random algorithm changing: https://bugs.python.org/issue27742

This is clear from examining a diff of the descriptions generated - the only difference is os, which is chosen by random.randint(). Because of this, using the same seed with python3 has different results than suites that were scheduled with python2. There are only so many suites scheduled with py2, so I'm not sure it's worth changing anything here.

New runs with py3 will work with --rerun.

For any existing py2 runs, you can use teuthology-suite with python2 and --rerun will work.

Since python3 is the default in teuthology now, there shouldn't be many new runs scheduled with python2 in the future, so in a few weeks you shouldn't have to think about using python3 or python2.

Actions
Copy link
 #9

Updated by Kyrylo Shatskyy about 4 years ago

Josh Durgin wrote:

The root cause is the random algorithm changing: https://bugs.python.org/issue27742

This is clear from examining a diff of the descriptions generated - the only difference is os, which is chosen by random.randint(). Because of this, using the same seed with python3 has different results than suites that were scheduled with python2. There are only so many suites scheduled with py2, so I'm not sure it's worth changing anything here.

New runs with py3 will work with --rerun.

For any existing py2 runs, you can use teuthology-suite with python2 and --rerun will work.

Since python3 is the default in teuthology now, there shouldn't be many new runs scheduled with python2 in the future, so in a few weeks you shouldn't have to think about using python3 or python2.

that makes sense to me. so, close the ticket?

Actions
Copy link
 #10

Updated by Josh Durgin about 4 years ago

  • Status changed from New to Won't Fix
Actions
Copy link
 #11

Updated by Nathan Cutler about 4 years ago

  • Has duplicate Bug #35951: Recently merged "$" feature broke --filter and --rerun added
Actions
Copy link

Also available in: Atom PDF