Project

General

Profile

Actions

Bug #14336

closed

All runs failing in ovh lab

Added by Yuri Weinstein over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

See last 2 days runs, here is an example:
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-10_18:02:01-rgw-hammer-distro-basic-openstack/
Jobs - all
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-01-10_18:02:01-rgw-hammer-distro-basic-openstack/1000/teuthology.log

ubuntu-14.04' --flavor 'vps-ssd-3' --key-name teuthology  --user-data /tmp/tmpGrJUte  --min 2 --max 2 --security-group teuthology --property teuthology=31732c341c3ab9b553764d332a95d041 --property ownedby=149.202.184.32 --wait  target
2016-01-10T18:15:15.990 DEBUG:teuthology.misc:flock: --timeout: No such file or directory
2016-01-10T18:15:15.991 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/internal.py", line 114, in lock_machines
    ctx.archive, os_type, os_version, arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 411, in lock_many
    arch=arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 384, in lock_many_openstack
    num, os_type, os_version, arch, resources_hint)
  File "/home/teuthworker/src/teuthology_master/teuthology/provision.py", line 325, in create
    " " + self.basename)
  File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 1326, in sh
    output=output
CalledProcessError: Command 'flock --close /tmp/teuthology-server-create.lock --timeout 28800 openstack server create  -f json  --image 'teuthology-ubuntu-14.04' --flavor 'vps-ssd-3' --key-name teuthology  --user-data /tmp/tmpGrJUte  --min 2 --max 2 --security-group teuthology --property teuthology=31732c341c3ab9b553764d332a95d041 --property ownedby=149.202.184.32 --wait  target' returned non-zero exit status 69
2016-01-10T18:15:16.008 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=ac1596c2d9474bdcad07280e376dcd0c
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/internal.py", line 114, in lock_machines
    ctx.archive, os_type, os_version, arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 411, in lock_many
    arch=arch)
  File "/home/teuthworker/src/teuthology_master/teuthology/lock.py", line 384, in lock_many_openstack
    num, os_type, os_version, arch, resources_hint)
  File "/home/teuthworker/src/teuthology_master/teuthology/provision.py", line 325, in create
    " " + self.basename)
  File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 1326, in sh
    output=output
CalledProcessError: Command 'flock --close /tmp/teuthology-server-create.lock --timeout 28800 openstack server create  -f json  --image 'teuthology-ubuntu-14.04' --flavor 'vps-ssd-3' --key-name teuthology  --user-data /tmp/tmpGrJUte  --min 2 --max 2 --security-group teuthology --property teuthology=31732c341c3ab9b553764d332a95d041 --property ownedby=149.202.184.32 --wait  target' returned non-zero exit status 69
2016-01-10T18:15:16.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2016-01-10T18:15:16.008 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CalledProcessError: Command 'flock --close /tmp/teuthology-server-create.lock --timeout 28800 openstack server create  -f json  --image 'teuthology-ubuntu-14.04' --flavor 'vps-ssd-3' --key-name teuthology  --user-data /tmp/tmpGrJUte  --min 2 --max 2 --security-group teuthology --property teuthology=31732c341c3ab9b553764d332a95d041 --property ownedby=149.202.184.32 --wait  target' returned non-zero exit status 69
2016-01-10T18:15:16.011 INFO:teuthology.run:Summary data:
{description: 'rgw/verify/{overrides.yaml clusters/fixed-2.yaml frontend/civetweb.yaml
    fs/btrfs.yaml msgr-failures/few.yaml rgw_pool_type/ec-profile.yaml tasks/rgw_swift.yaml
    validater/lockdep.yaml}', failure_reason: Command 'flock --close /tmp/teuthology-server-create.lock
    --timeout 28800 openstack server create  -f json  --image 'teuthology-ubuntu-14.04'
    --flavor 'vps-ssd-3' --key-name teuthology  --user-data /tmp/tmpGrJUte  --min
    2 --max 2 --security-group teuthology --property teuthology=31732c341c3ab9b553764d332a95d041
    --property ownedby=149.202.184.32 --wait  target' returned non-zero exit status
    69, owner: scheduled_teuthology@teuthology, sentry_event: 'http://sentry.ceph.com/sepia/teuthology/?q=ac1596c2d9474bdcad07280e376dcd0c',
  status: fail, success: false}

Actions #1

Updated by Zack Cerza over 8 years ago

  • Status changed from New to In Progress
  • Assignee set to Zack Cerza
Actions #2

Updated by Zack Cerza over 8 years ago

zack@teuthology:~$ flock --close /tmp/zlock echo hi
hi
zack@teuthology:~$ flock --close /tmp/zlock --timeout 1 echo hi
flock: --timeout: No such file or directory
zack@teuthology:~$ flock --close --timeout 1 /tmp/zlock echo hi
hi

This is a regression: https://github.com/ceph/teuthology/pull/764
https://github.com/ceph/teuthology/pull/764/files#diff-1617d70bcf3cfa797ee2aec9597d787dR310

Actions #3

Updated by Zack Cerza over 8 years ago

  • Status changed from In Progress to Fix Under Review
Actions #4

Updated by Zack Cerza over 8 years ago

  • Status changed from Fix Under Review to Resolved
Actions #5

Updated by Zack Cerza over 8 years ago

  • Status changed from Resolved to Fix Under Review
Actions #6

Updated by Zack Cerza over 8 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF