CI

teuthology@teuthology:~$ CEPH_BRANCH=master; MACHINE_NAME=ovh; teuthology-suite -v -c $CEPH_BRANCH -n 7 -m $MACHINE_NAME -s smoke -k testing -p 70 -e $CEPH_QA_EMAIL
2018-06-21 15:22:41,592.592 INFO:teuthology.suite:Using random seed=9723
2018-06-21 15:22:41,593.593 DEBUG:teuthology.suite.util:Defaults for machine_type ovh distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2018-06-21 15:22:41,594.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F7%2Fx86_64&ref=testing
2018-06-21 15:22:41,977.977 INFO:teuthology.suite.run:kernel sha1: d31fd253bd4c9219df45b98ab31598c2ac540abf
2018-06-21 15:22:51,288.288 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph master -> 975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:22:51,288.288 INFO:teuthology.suite.run:ceph sha1: 975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:22:51,289.289 INFO:teuthology.suite.run:skipping ceph package verification
2018-06-21 15:22:51,289.289 INFO:teuthology.suite.run:teuthology branch: master
2018-06-21 15:23:00,147.147 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph.git master -> 975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:23:08,766.766 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph.git master -> 975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:23:08,766.766 INFO:teuthology.suite.run:ceph branch: master 975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:23:08,770.770 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/teuthology.git
2018-06-21 15:23:08,776.776 INFO:teuthology.repo_utils:Fetching master from upstream
2018-06-21 15:23:08,851.851 INFO:teuthology.repo_utils:Resetting repo at /home/teuthology/src/git.ceph.com_git_teuthology_master to branch master
2018-06-21 15:23:08,864.864 INFO:teuthology.repo_utils:Bootstrapping /home/teuthology/src/git.ceph.com_git_teuthology_master
2018-06-21 15:23:18,019.019 INFO:teuthology.repo_utils:Bootstrap exited with status 0
2018-06-21 15:23:18,026.026 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/ceph.git
2018-06-21 15:23:18,031.031 INFO:teuthology.repo_utils:Fetching master from upstream
2018-06-21 15:23:28,007.007 INFO:teuthology.repo_utils:Resetting repo at /home/teuthology/src/git.ceph.com_ceph_master to branch master
2018-06-21 15:23:28,121.121 DEBUG:teuthology.suite.run:Suite smoke in /home/teuthology/src/git.ceph.com_ceph_master/qa/suites/smoke
2018-06-21 15:23:28,127.127 INFO:teuthology.suite.run:Suite smoke in /home/teuthology/src/git.ceph.com_ceph_master/qa/suites/smoke generated 28 jobs (not yet filtered)
2018-06-21 15:23:28,139.139 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:23:28,139.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=975736aba546adb09ce4f38a1ae061621f7240cd
2018-06-21 15:23:28,379.379 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash '975736aba546adb09ce4f38a1ae061621f7240cd' not found
2018-06-21 15:23:38,149.149 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:23:38,149.149 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=c37c5dd43e618da65cac4d67da81a158783d074b
2018-06-21 15:23:38,362.362 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'c37c5dd43e618da65cac4d67da81a158783d074b' not found
2018-06-21 15:23:48,517.517 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:23:48,518.518 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=679ce7a35208ddb47197e5fc2dad7ed6cf695770
2018-06-21 15:23:48,740.740 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash '679ce7a35208ddb47197e5fc2dad7ed6cf695770' not found
2018-06-21 15:23:59,270.270 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:23:59,271.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=80a51c9714a621a5bc6bb05b8e15244fd43994ec
2018-06-21 15:23:59,513.513 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash '80a51c9714a621a5bc6bb05b8e15244fd43994ec' not found
2018-06-21 15:24:09,026.026 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:24:09,027.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=96028cf94db9f9c4ce21936ac947f89dc2fed1ca
2018-06-21 15:24:09,268.268 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash '96028cf94db9f9c4ce21936ac947f89dc2fed1ca' not found
2018-06-21 15:24:19,378.378 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:24:19,378.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=e737fbe603f6da8114f1f79b2baa8fc57c569cd1
2018-06-21 15:24:19,616.616 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'e737fbe603f6da8114f1f79b2baa8fc57c569cd1' not found
2018-06-21 15:24:29,602.602 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:24:29,603.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=6487184ebbd0669abb2028fe59490fd41dfa99bc
2018-06-21 15:24:29,834.834 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash '6487184ebbd0669abb2028fe59490fd41dfa99bc' not found
2018-06-21 15:24:41,053.053 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/14.04, pkg_type=deb
2018-06-21 15:24:41,054.054 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=af83759a47e4a60cc7b437355c7c23c430acb5cf
2018-06-21 15:24:41,295.295 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'af83759a47e4a60cc7b437355c7c23c430acb5cf' not found
Traceback (most recent call last):
  File "/home/teuthology/src/teuthology_master/virtualenv/bin/teuthology-suite", line 11, in <module>
    load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')()
  File "/home/teuthology/src/teuthology_master/scripts/suite.py", line 142, in main
    return teuthology.suite.main(args)
  File "/home/teuthology/src/teuthology_master/teuthology/suite/__init__.py", line 92, in main
    run.prepare_and_schedule()
  File "/home/teuthology/src/teuthology_master/teuthology/suite/run.py", line 309, in prepare_and_schedule
    num_jobs = self.schedule_suite()
  File "/home/teuthology/src/teuthology_master/teuthology/suite/run.py", line 514, in schedule_suite
    name,
  File "/home/teuthology/src/teuthology_master/teuthology/suite/util.py", line 72, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling teuthology-2018-06-21_15:22:41-smoke-master-testing-basic-ovh failed: Exceeded 7 backtracks; raise --newest value

Some chatter from IRC

(08:27:56 AM) yuriw: we have runs failing to schedule, one example https://pastebin.com/nyfmyk8A I am not sure why even though master, mimic seem all green  https://shaman.ceph.com/builds/ceph/master/
(08:27:56 AM) yuriw: any clues wth ?
(08:27:56 AM) yuriw: sage zackc alfredodeza dgalloway ^
(08:29:04 AM) dgalloway: is mimic built for trusty?
(08:29:13 AM) alfredodeza: yuriw: that sha1 is not available https://shaman.ceph.com/builds/ceph/master/af83759a47e4a60cc7b437355c7c23c430acb5cf/
(08:29:16 AM) alfredodeza: it has never been built
(08:29:26 AM) alfredodeza: hence the: 2018-06-21 15:24:41,295.295 ERROR:teuthology.suite.run:Packages for os_type 'ubuntu', flavor basic and ceph hash 'af83759a47e4a60cc7b437355c7c23c430acb5cf' not found
(08:29:37 AM) yuriw: but why  ?
(08:30:11 AM) yuriw: the line is => CEPH_BRANCH=master; MACHINE_NAME=ovh; teuthology-suite -v -c $CEPH_BRANCH -n 7 -m $MACHINE_NAME -s smoke -k testing -p 70 -e $CEPH_QA_EMAIL
(08:42:44 AM) zackc: suggests to me that builds for master are >7 commits behind for ubuntu
(08:43:21 AM) zackc: not terribly surprising though; i think i've seen a few conversations lately about long build times because of debug packages growing in size
(08:44:06 AM) alfredodeza: that doesn't sound right though, that sha1 is from 20 hours ago
(08:44:25 AM) alfredodeza: build times take about 1.5 hours
(08:46:58 AM) zackc: hm
(08:49:15 AM) zackc: alfredodeza: is the list of sha1s at https://shaman.ceph.com/builds/ceph/master/ in order?
(08:50:26 AM) zackc: doesn't appear so
(08:52:23 AM) zackc: would be so helpful here if we had a view that could visualize commits on a branch alongside shaman's availability for each commit
(08:52:40 AM) zackc: i do have some understanding of why we don't have that, though :)
(08:53:32 AM) alfredodeza: I don't think that the sha1s are in order no (assuming you mean of commit order as it appears in git)
(08:53:41 AM) alfredodeza: they are in order of being built I think
(08:54:27 AM) zackc: right, that's what i meant
(08:55:32 AM) alfredodeza: for sure it is not using git order. But I am not certain if it is following build order or 'last modified' order
(08:55:48 AM) alfredodeza: there are a few states a build process goes which changes the 'last modified' flag
(08:56:09 AM) alfredodeza: I don't think I've seen this code in over a year, so there is plenty of room for being very wrong 
....
(09:14:31 AM) alfredodeza: I don't know why that sha1 is not available yuriw