Tools » teuthology

here I created basic yaml file to run some tasks on some nodes. Connections are done from admin to clients but unable to clear lock servers.

how do I solve this issue.

here attached my example.yaml file:
-------------------------------------

check-locks: false
roles:
- [mon.0, mds.0, osd.0]
- [mon.1, osd.1]
- [mon.2, client.0]

tasks:
- install:
project: ceph
branch: stable
- ceph:
- kclient: [client.0]
- interactive:

targets:
sri-target1: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ+8MHeLNyaDqdV5dROV00VcoBkLkriqpu2h2cKfOEdEYQlD0Hflxb5P2QJmX4Wm/5nyfEXuyp8MCwL7uHrb+YWXRfFGo1l0VUldwrG5NeJcsVgWNpyYvyhAckQ3vHFUUYpLRWgWqHRUIZF+feZd28mEPpew9/OPphF6O7hwT7kwri3E/2L8VdzxnX2JLhrw0f0unK9wQzE5/dA8KXxyAz/oXZda7JmYR0Qgdw2M/6HSdSVTAFC/v1pof9Jjxz9tsnnhpWnpIFZsyalX3UVVIpIde1oA2MygEWuxAoR9bLrxC0iOMlxxe6wXvblduqa4kdmZh5loM0xrAOrodmMGMh

sri-target2: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoi5fwsIm5Gvg5w8eUND6lPlOOyc4qYDgMNPnkOoNiY93WArD643R45NRgMzndvAfDj2CKSrDL4WjE1mgG7xBS33EBsiKyerLhlr+8j03z/4iKP2q5boJYnRvN//Ky9Mzduyc9BxPCf5PSefBttnJEWl2rjL29RO+hac54MJnR9/Laq4s3owExMo/fNJOoMxM2iZ1bB28esBA0vDfudget3BtvkCnZ/e029VvnvgjtGgEVloMT6UvElMKhZSO4Dxf7Es5hQBdDroYBKqOgQhQFd/kFwLxxJlZc4rYJNygupfhDpzI/VzeZ8uIuH9PkEmVjNNh014eBCo1QQnmFXMMr

sri-target3: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMNA9v1byUfy43/O5abEyUuvjn+qLOO20TDP0OkzEOhwmlxZkerVgfPaZb2TaYme6xPYQfKt55SKYFZl86XVGJljlsCezkh6hZ3f4zGDyHSDJR9HkMWAG7QYxx4rumNq15l87O6UasIbxN84rTVUGmAyoua2HJe9QqqbgomeEjaqzK510SzdPw4CLTOvlI19uWc6vIbCEn40h7la49yAj3u5w0cDINOxJE/6sgwZlSA1vK7XiXzRF7kZOii7AabIiwR/417iwDMW+X4F87TOqdHvrLrKd6Rg7DYADyxGKZaLDsHRd2LaJ4sJGL4itad3wGWPTHNGhQjAx8vEWu+w+l

run command:
-----------

$./virtualenv/bin/teuthology example.yaml

error: ======

2015-07-23 09:12:43,856.856 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-23 09:12:43,857.857 WARNING:teuthology.run:No tasks specified. Continuing anyway...
2015-07-23 09:12:43,857.857 INFO:teuthology.run:Tasks not found; will attempt to fetch
2015-07-23 09:12:43,857.857 INFO:teuthology.repo_utils:Fetching from upstream into /home/ubuntu/src/ceph-qa-suite_master
2015-07-23 09:12:51,789.789 INFO:teuthology.repo_utils:Resetting repo at /home/ubuntu/src/ceph-qa-suite_master to branch master
2015-07-23 09:12:51,806.806 INFO:teuthology.run_tasks:Running task internal.check_packages...
2015-07-23 09:12:51,807.807 INFO:teuthology.task.internal:Checking packages...
2015-07-23 09:12:51,807.807 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash 'None'
2015-07-23 09:12:51,807.807 INFO:teuthology.run_tasks:Running task internal.save_config...
2015-07-23 09:12:51,807.807 INFO:teuthology.task.internal:Saving configuration
2015-07-23 09:12:51,807.807 INFO:teuthology.run_tasks:Running task internal.check_lock...
2015-07-23 09:12:51,808.808 INFO:teuthology.task.internal:Checking locks...
2015-07-23 09:12:51,808.808 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
return fn(**kwargs)
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 209, in check_lock
for machine in ctx.config['targets'].iterkeys():
KeyError: 'targets'
2015-07-23 09:12:51,810.810 INFO:teuthology.run:Summary data: {failure_reason: '''targets''', owner: ubuntu@teuthology, status: fail, success: false}

2015-07-23 09:12:51,811.811 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-23 09:12:51,811.811 INFO:teuthology.run:FAIL
ubuntu@teuthology:~/teuthology$ ^C
ubuntu@teuthology:~/teuthology$ ./virtualenv/bin/teuthology examples/3node_ceph.yaml
2015-07-23 09:13:02,138.138 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-23 09:13:02,143.143 INFO:teuthology.run:Tasks not found; will attempt to fetch
2015-07-23 09:13:02,144.144 INFO:teuthology.repo_utils:Fetching from upstream into /home/ubuntu/src/ceph-qa-suite_master
2015-07-23 09:13:03,805.805 INFO:teuthology.repo_utils:Resetting repo at /home/ubuntu/src/ceph-qa-suite_master to branch master
2015-07-23 09:13:03,822.822 INFO:teuthology.run_tasks:Running task internal.check_packages...
2015-07-23 09:13:03,822.822 INFO:teuthology.task.internal:Checking packages...
2015-07-23 09:13:03,823.823 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash 'None'
2015-07-23 09:13:03,823.823 INFO:teuthology.run_tasks:Running task internal.save_config...
2015-07-23 09:13:03,823.823 INFO:teuthology.task.internal:Saving configuration
2015-07-23 09:13:03,823.823 INFO:teuthology.run_tasks:Running task internal.check_lock...
2015-07-23 09:13:03,824.824 INFO:teuthology.task.internal:Lock checking disabled.
2015-07-23 09:13:03,824.824 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2015-07-23 09:13:03,825.825 INFO:teuthology.task.internal:roles: ubuntu@sri-target1.front.sepia.ceph.com - ['mon.0', 'mds.0', 'osd.0']
2015-07-23 09:13:03,825.825 INFO:teuthology.task.internal:roles: ubuntu@sri-target3.front.sepia.ceph.com - ['mon.1', 'osd.1']
2015-07-23 09:13:03,825.825 INFO:teuthology.task.internal:roles: ubuntu@sri-target2.front.sepia.ceph.com - ['mon.2', 'client.0']
2015-07-23 09:13:03,825.825 INFO:teuthology.run_tasks:Running task internal.connect...
2015-07-23 09:13:03,826.826 INFO:teuthology.task.internal:Opening connections...
2015-07-23 09:13:03,826.826 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'sri-target3.front.sepia.ceph.com', 'timeout': 60}
2015-07-23 09:13:04,251.251 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'sri-target1.front.sepia.ceph.com', 'timeout': 60}
2015-07-23 09:13:04,691.691 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'sri-target2.front.sepia.ceph.com', 'timeout': 60}
2015-07-23 09:13:05,145.145 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2015-07-23 09:13:05,145.145 INFO:teuthology.orchestra.run.sri-target3:Running: 'uname -m'
2015-07-23 09:13:06,731.731 INFO:teuthology.orchestra.run.sri-target3:Running: "python -c 'import platform; print platform.linux_distribution()'"
2015-07-23 09:13:06,824.824 INFO:teuthology.lock:Updating sri-target3.front.sepia.ceph.com on lock server
2015-07-23 09:15:16,933.933 ERROR:teuthology.task.internal:Error pushing inventory
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 335, in push_inventory
push()
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 333, in push
lock.update_inventory(info)
File "/home/ubuntu/teuthology/teuthology/lock.py", line 636, in update_inventory
headers={'content-type': 'application/json'},
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 122, in put
return request('put', url, data=data, *kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
2015-07-23 09:15:16,938.938 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2015-07-23 09:15:16,939.939 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2015-07-23 09:15:16,940.940 INFO:teuthology.task.internal:Checking for old test directory...
2015-07-23 09:15:16,940.940 INFO:teuthology.orchestra.run.sri-target1:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-23 09:15:16,945.945 INFO:teuthology.orchestra.run.sri-target2:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-23 09:15:16,950.950 INFO:teuthology.orchestra.run.sri-target3:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-23 09:15:16,958.958 ERROR:teuthology.task.internal:Host sri-target1 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2015-07-23 09:15:16,958.958 ERROR:teuthology.task.internal:Host sri-target2 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2015-07-23 09:15:16,962.962 ERROR:teuthology.task.internal:Host sri-target3 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2015-07-23 09:15:16,963.963 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
return fn(*kwargs)
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 394, in check_conflict
raise RuntimeError('Stale jobs detected, aborting.')
RuntimeError: Stale jobs detected, aborting.
2015-07-23 09:15:16,966.966 INFO:teuthology.run:Summary data: {failure_reason: 'Stale jobs detected, aborting.', owner: ubuntu@teuthology, status: fail,
success: false}

2015-07-23 09:15:16,966.966 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-23 09:15:16,966.966 INFO:teuthology.run:FAIL