@zackc here is a case:
teuthology@teuthology:~$ echo $filter
upgrade:hammer-x/f-h-x-offline/1-pre.yaml,upgrade:hammer-x/f-h-x-offline/2-upgrade.yaml,upgrade:hammer-x/f-h-x-offline/3-infernalis.yaml,upgrade:hammer-x/f-h-x-offline/4-after.yaml
teuthology@teuthology:~$ teuthology-suite -v -c infernalis -k distro -m vps -s upgrade/hammer-x ~/vps.yaml -p 100 --filter="$filter"
2015-11-04 16:16:10,588.588 INFO:teuthology.suite:kernel sha1: distro
2015-11-04 16:16:10,588.588 DEBUG:teuthology.suite:Defaults for machine_type vps distro None: arch=x86_64, release=centos7, pkg_type=rpm
2015-11-04 16:16:10,588.588 DEBUG:teuthology.suite:Gitbuilder URL: http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/infernalis/sha1
2015-11-04 16:16:10,607.607 INFO:teuthology.suite:ceph sha1: bb2ecea240f3a1d525bcb35670cb07bd1f0ca299
2015-11-04 16:16:10,607.607 DEBUG:teuthology.suite:Defaults for machine_type vps distro None: arch=x86_64, release=centos7, pkg_type=rpm
2015-11-04 16:16:10,607.607 DEBUG:teuthology.suite:Looking for packages at http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/sha1/bb2ecea240f3a1d525bcb35670cb07bd1f0ca299/version
2015-11-04 16:16:10,611.611 INFO:teuthology.suite:ceph version: v9.2.0
2015-11-04 16:16:10,611.611 INFO:teuthology.suite:teuthology branch: master
2015-11-04 16:16:11,938.938 INFO:teuthology.suite:ceph-qa-suite branch: infernalis f9e22c5e046d032a6a52b4530ae225752a6ed085
2015-11-04 16:16:11,939.939 INFO:teuthology.repo_utils:Fetching from upstream into /home/teuthology/src/teuthology_master
2015-11-04 16:16:12,609.609 INFO:teuthology.repo_utils:Resetting repo at /home/teuthology/src/teuthology_master to branch master
2015-11-04 16:16:12,616.616 INFO:teuthology.repo_utils:Bootstrapping /home/teuthology/src/teuthology_master
2015-11-04 16:16:20,197.197 INFO:teuthology.repo_utils:Bootstrap exited with status 0
2015-11-04 16:16:20,198.198 INFO:teuthology.repo_utils:Fetching from upstream into /home/teuthology/src/ceph-qa-suite_infernalis
2015-11-04 16:16:21,520.520 INFO:teuthology.repo_utils:Resetting repo at /home/teuthology/src/ceph-qa-suite_infernalis to branch infernalis
2015-11-04 16:16:21,565.565 DEBUG:teuthology.suite:Suite upgrade:hammer-x in /home/teuthology/src/ceph-qa-suite_infernalis/suites/upgrade/hammer-x
2015-11-04 16:16:21,572.572 INFO:teuthology.suite:Suite upgrade:hammer-x in /home/teuthology/src/ceph-qa-suite_infernalis/suites/upgrade/hammer-x generated 20 jobs (not yet filtered)
2015-11-04 16:16:21,580.580 INFO:teuthology.suite:Scheduling upgrade:hammer-x/f-h-x-offline/1-pre.yaml
Job scheduled with name teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps and ID 1138893
2015-11-04 16:16:22,174.174 INFO:teuthology.suite:Scheduling upgrade:hammer-x/f-h-x-offline/2-upgrade.yaml
Job scheduled with name teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps and ID 1138894
2015-11-04 16:16:22,674.674 INFO:teuthology.suite:Scheduling upgrade:hammer-x/f-h-x-offline/3-infernalis.yaml
Job scheduled with name teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps and ID 1138895
2015-11-04 16:16:23,188.188 INFO:teuthology.suite:Scheduling upgrade:hammer-x/f-h-x-offline/4-after.yaml
Job scheduled with name teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps and ID 1138896
2015-11-04 16:16:23,663.663 INFO:teuthology.suite:Suite upgrade:hammer-x in /home/teuthology/src/ceph-qa-suite_infernalis/suites/upgrade/hammer-x scheduled 4 jobs.
2015-11-04 16:16:23,663.663 INFO:teuthology.suite:Suite upgrade:hammer-x in /home/teuthology/src/ceph-qa-suite_infernalis/suites/upgrade/hammer-x -- 16 jobs were filtered out.
Job scheduled with name teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps and ID 1138897
2015-11-04 16:16:24,110.110 INFO:teuthology.suite:Test results viewable at http://pulpito.ceph.com/teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps/
In pulpito: http://pulpito.ceph.com/teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps/
All 'queued' and resulting email came empty of cause, subject "[Ceph-qa] in teuthology-2015-11-04_16:16:10-upgrade:hammer-x-infernalis-distro-basic-vps"