Infrastructure

Note from Zack regarding this issue:

looking at paddles logs again, I see this:
Feb 09 16:35:16 pulpito sudo[32458]:     root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/docker exec -it fa31f1474a1a sh -c pecan expire_jobs config.py -q 0 -r 600
and I am pretty sure the -q 0 there would cause all queued jobs to be marked dead in paddles - but the expire_jobs tool does not touch the beanstalkd queue, which is the source of truth. this is a bizarre situation, but i believe those jobs are actually still queued and will run eventually

Also, teuthology-kill only touches jobs that are actively running or in the 'queued' state. teuthology-queue has more flexible options for dequeueing jobs.

Zack Cerza wrote:

Also, teuthology-kill only touches jobs that are actively running or in the 'queued' state. teuthology-queue has more flexible options for dequeueing jobs.

Thanks Zack, I didn't know this was an option!

teuthology-queue options for those interested:

lflores@teuthology:~$ ./teuthology/virtualenv/bin/teuthology-queue --help
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend
usage: teuthology-queue -h
       teuthology-queue [-s|-d|-f] -m MACHINE_TYPE
       teuthology-queue [-r] -m MACHINE_TYPE
       teuthology-queue -m MACHINE_TYPE -D PATTERN
       teuthology-queue -p SECONDS [-m MACHINE_TYPE]

List Jobs in queue.
If -D is passed, then jobs with PATTERN in the job name are deleted from the
queue.

Arguments:
  -m, --machine_type MACHINE_TYPE [default: multi]
                        Which machine type queue to work on.

optional arguments:
  -h, --help            Show this help message and exit
  -D, --delete PATTERN  Delete Jobs with PATTERN in their name
  -d, --description     Show job descriptions
  -r, --runs            Only show run names
  -f, --full            Print the entire job config. Use with caution.
  -s, --status          Prints the status of the queue
  -p, --pause SECONDS   Pause queues for a number of seconds. A value of 0
                        will unpause. If -m is passed, pause that queue,
                        otherwise pause all queues.