Project

General

Profile

Bug #19694

teuthology-suite allows partial schedules on complete (incomplete?) builds

Added by Yuri Weinstein almost 7 years ago. Updated almost 7 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Crash signature (v1):
Crash signature (v2):

Description

Trying to schedule a run on seemly complete sha1 https://shaman.ceph.com/builds/ceph/wip-sage-testing/cc116d47cab9be2b8f23de47cdfd39b9b0994234/

 yuriw@teuthology ~ [23:54:57]> CEPH_QA_MAIL="ceph-qa@ceph.com"; MACHINE_NAME=smithi; CEPH_BRANCH=wip-sage-testing
  yuriw@teuthology ~ [16:30:42]> 
  yuriw@teuthology ~ [16:30:43]> 
  yuriw@teuthology ~ [16:30:43]> 
  yuriw@teuthology ~ [16:30:43]> 
  yuriw@teuthology ~ [16:30:44]> teuthology-suite -v --ceph-repo https://github.com/ceph/ceph-ci.git -c $CEPH_BRANCH -m $MACHINE_NAME -s rados --subset 35/350 -k distro -p 100 -e $CEPH_QA_MAIL -S cc116d47cab9be2b8f23de47cdfd39b9b0994234;
2017-04-19 16:31:24,149.149 INFO:teuthology.suite.run:kernel sha1: distro
2017-04-19 16:31:24,692.692 INFO:teuthology.suite.run:ceph sha1 explicitly supplied
2017-04-19 16:31:24,692.692 INFO:teuthology.suite.run:ceph sha1: cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:24,692.692 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2017-04-19 16:31:24,693.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:24,886.886 INFO:teuthology.suite.run:ceph version: 12.0.0-2837.gcc116d4
2017-04-19 16:31:24,886.886 INFO:teuthology.suite.run:teuthology branch: master
2017-04-19 16:31:24,970.970 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci.git wip-sage-testing -> cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:24,989.989 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci.git wip-sage-testing -> cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:24,989.989 INFO:teuthology.suite.run:ceph-ci branch: wip-sage-testing cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:25,211.211 INFO:teuthology.repo_utils:Cloning git://git.ceph.com/ceph-ci.git wip-sage-testing from upstream
2017-04-19 16:31:28,805.805 INFO:teuthology.repo_utils:Resetting repo at /home/yuriw/src/git.ceph.com_ceph-c_wip-sage-testing to branch wip-sage-testing
2017-04-19 16:31:29,143.143 DEBUG:teuthology.suite.run:Suite rados in /home/yuriw/src/git.ceph.com_ceph-c_wip-sage-testing/qa/suites/rados
2017-04-19 16:31:29,143.143 INFO:teuthology.suite.build_matrix:Subset=35/350
2017-04-19 16:31:29,448.448 INFO:teuthology.suite.run:Suite rados in /home/yuriw/src/git.ceph.com_ceph-c_wip-sage-testing/qa/suites/rados generated 292 jobs (not yet filtered)
2017-04-19 16:31:29,499.499 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2017-04-19 16:31:29,500.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:30,427.427 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2017-04-19 16:31:30,427.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:31,391.391 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2017-04-19 16:31:31,391.391 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:31,914.914 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2017-04-19 16:31:31,915.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:32,502.502 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2017-04-19 16:31:32,502.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:32,698.698 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'cc116d47cab9be2b8f23de47cdfd39b9b0994234' not found
2017-04-19 16:31:34,692.692 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2017-04-19 16:31:34,692.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=cc116d47cab9be2b8f23de47cdfd39b9b0994234
2017-04-19 16:31:34,883.883 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'cc116d47cab9be2b8f23de47cdfd39b9b0994234' not found
2017-04-19 16:31:35,067.067 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/peering_and_degraded.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml thrashers/none.yaml workloads/cache-agent-big.yaml z-require-luminous/at-mkfs.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044549
2017-04-19 16:31:36,555.555 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/normal.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/pggrow.yaml workloads/cache-agent-small.yaml z-require-luminous/at-end.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044550
2017-04-19 16:31:37,251.251 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/peering.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-btrfs.yaml rados.yaml rocksdb.yaml thrashers/default.yaml workloads/cache-pool-snaps-readproxy.yaml z-require-luminous/at-mkfs.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044551
2017-04-19 16:31:37,898.898 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_degraded.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml thrashers/mapgap.yaml workloads/cache-pool-snaps.yaml z-require-luminous/at-end.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044552
2017-04-19 16:31:38,557.557 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/normal.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/morepggrow.yaml workloads/cache-snaps.yaml z-require-luminous/at-mkfs.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044553
2017-04-19 16:31:39,238.238 INFO:teuthology.suite.run:Scheduling rados/objectstore/objectstore.yaml
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044554
2017-04-19 16:31:39,954.954 INFO:teuthology.suite.run:Scheduling rados/singleton/{all/rebuild-mondb.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044555
2017-04-19 16:31:40,667.667 INFO:teuthology.suite.run:Scheduling rados/thrash-erasure-code-shec/{clusters/{fixed-4.yaml openstack.yaml} leveldb.yaml msgr-failures/fastclose.yaml objectstore/filestore-btrfs.yaml rados.yaml thrashers/default.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044556
....
....
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044679
2017-04-19 16:33:00,806.806 INFO:teuthology.suite.run:Scheduling rados/thrash-erasure-code-big/{cluster/{12-osds.yaml openstack.yaml} leveldb.yaml msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044680
2017-04-19 16:33:01,516.516 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/normal.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/default.yaml workloads/cache-snaps.yaml z-require-luminous/at-mkfs.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044681
2017-04-19 16:33:02,146.146 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-btrfs.yaml rados.yaml rocksdb.yaml thrashers/mapgap.yaml workloads/cache.yaml z-require-luminous/at-end.yaml}
Job scheduled with name yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi and ID 1044682
2017-04-19 16:33:02,854.854 INFO:teuthology.suite.run:Scheduling rados/thrash-erasure-code-isa/{arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} leveldb.yaml msgr-failures/few.yaml objectstore/bluestore.yaml rados.yaml supported/ubuntu_latest.yaml thrashers/none.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}
Traceback (most recent call last):
  File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 11, in <module>
    load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')()
  File "/home/yuriw/teuthology/scripts/suite.py", line 137, in main
    return teuthology.suite.main(args)
  File "/home/yuriw/teuthology/teuthology/suite/__init__.py", line 88, in main
    run.prepare_and_schedule()
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 309, in prepare_and_schedule
    num_jobs = self.schedule_suite()
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 525, in schedule_suite
    self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 447, in schedule_jobs
    name,
  File "/home/yuriw/teuthology/teuthology/suite/util.py", line 72, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi failed: At least one job needs packages that don't exist for hash cc116d47cab9be2b8f23de47cdfd39b9b0994234.

result in pulpito => http://pulpito.ceph.com/yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi/

and on top of all failed email was delievred

Subject: [Ceph-qa] Failed to schedule yuriw-2017-04-19_16:31:24-rados-wip-sage-testing-distro-basic-smithi
To: ceph-qa@ceph.com

At least one job needs packages that don't exist for hash cc116d47cab9be2b8f23de47cdfd39b9b0994234.
_______________________________________________
Ceph-qa mailing list
Ceph-qa@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-qa-ceph.com

History

#1 Updated by Yuri Weinstein almost 7 years ago

  • Subject changed from @teuthology-suite@ allows partial schedules on incomplete builds to @teuthology-suite@ allows partial schedules on complete (incomplete?) builds

#2 Updated by Yuri Weinstein almost 7 years ago

  • Subject changed from @teuthology-suite@ allows partial schedules on complete (incomplete?) builds to teuthology-suite allows partial schedules on complete (incomplete?) builds

#3 Updated by Yuri Weinstein almost 7 years ago

teuthology-suite -v --ceph-repo https://github.com/ceph/ceph-ci.git -c $CEPH_BRANCH -m $MACHINE_NAME -s rados --subset 35/350 -k distro -p 100 -e $CEPH_QA_MAIL -S cc116d47cab9be2b8f23de47cdfd39b9b0994234 --dry-run

produced correct amount of jobs:

2017-04-19 16:42:12,644.644 INFO:teuthology.suite.run:Suite rados in /home/yuriw/src/git.ceph.com_ceph-c_wip-sage-testing/qa/suites/rados scheduled 292 jobs.

maybe shaman shows green status prematurely ? or user scheduled before something is not finished?

#4 Updated by Yuri Weinstein almost 7 years ago

  • Description updated (diff)

#5 Updated by Nathan Cutler almost 7 years ago

I ran into this today, too. I had been watching a Shaman build . . . it turned green so I tried to schedule a run on it. But somehow the packages had not propagated (or whatever) and this error was the result. Five minutes later I could schedule the run without issue.

Also available in: Atom PDF