Feature #14515: Failures to lock because of OpenStack quota issues should result in 'dead' jobs - teuthology - Ceph

Actions

Copy link

Feature #14515

closed

Failures to lock because of OpenStack quota issues should result in 'dead' jobs

Added by Zack Cerza over 8 years ago. Updated about 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Zack Cerza

Category:

% Done:

Source:

other

Tags:

Backport:

Reviewed:

Affected Versions:

Description

Currently, such jobs are failing. An example:
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-18_16:00:01-rbd-master---basic-openstack/6250

From the log:

2016-01-18T16:04:25.284 DEBUG:teuthology.misc::sh: flock --close --timeout 28800 /tmp/teuthology-server-create.lock openstack server create  -f json  --image 'teuthology-ubuntu-14.04' --flavor 'eg-30' --key-name teuthology  --user-data /tmp/tmpVeQTjr  --min 3 --max 3 --security-group teuthology --property teuthology=a4ab0be25258844081852308db3a820b --property ownedby=149.202.184.32 --wait  target
2016-01-18T16:13:53.884 DEBUG:teuthology.misc:Quota exceeded for ram: Requested 90000, but already used 4188000 of 4194304 ram (HTTP 403) (Request-ID: req-e31cc368-7541-48f7-96e3-60c5596035f0)
2016-01-18T16:13:53.919 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/internal.py", line 114, in lock_machines
    ctx.archive, os_type, os_version, arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 411, in lock_many
    arch=arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 384, in lock_many_openstack
    num, os_type, os_version, arch, resources_hint)
  File "/home/teuthworker/src/teuthology_master/teuthology/provision.py", line 325, in create
    " " + self.basename)
  File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 1326, in sh
    output=output
CalledProcessError: Command 'flock --close --timeout 28800 /tmp/teuthology-server-create.lock openstack server create  -f json  --image 'teuthology-ubuntu-14.04' --flavor 'eg-30' --key-name teuthology  --user-data /tmp/tmpVeQTjr  --min 3 --max 3 --security-group teuthology --property teuthology=a4ab0be25258844081852308db3a820b --property ownedby=149.202.184.32 --wait  target' returned non-zero exit status 1

I'm going to implement a separate exception for quota issues, and also generally make lock failures map to the 'dead' status instead of 'fail'.