Tools » teuthology

Hi Zack,

I have used flag 'use_existing_cluster' so that i could run teuthology on an exsisting cluster, but it failed giving error "Stale Job detected"

Teuthology log for your reference:

2014-10-21T10:58:57.289 WARNING:teuthology.report:No job_id found; not reporting results
2014-10-21T10:58:57.293 DEBUG:teuthology.run:Config:
  check-locks: false
  overrides:
    install:
      ceph:
        log-whitelist:
        - slow request
      local: /home/ems1/Sprint5eap2
  roles:
  - - mon.0
    - osd.0
    - osd.1
    - osd.2
  - - mon.1
    - osd.3
    - osd.4
    - osd.5
  - - mon.2
    - client.0
  targets:
    ubuntu@10.15.17.146: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCx4DupGNdntXrqfl6zjqpHAunfkrNkjijjWYFxH6gkCaguIGabZF0JO6nBDR7bM0h1mCVcSD6yzUHqA/TXuOuYPl/z4JZDm7EQoPfEsy5W+QUpqbAlwD370PhOGGN3oTVk7n7mktOyZ1iRaerPBGhWRHRythl6y2/gC/rrzgnP87pz563bO+IGYDOqlPJ0tKDCowsn/8cNQu/BlzMrIYDYnYperbg6+NbpFJfEiESM4aEeBE8MqIoHRznx1esYzW8bW+9SuncG2CNFwPPMeUasp69yYNUP+bLAnksC8VyNCsRTBqJmiEA7XZmsjh+i9BgGagp1oudhxIwRL6HcfeX7
    ubuntu@ip-10-15-16-182: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCcJGGh4XGVZE2Ol8412sId7tFig1tC0pbqiuYstsVLvXZci8oXRrH2boOPrdF7nxF2Uxx2rmw4b59UF38uqrUU+XuWnDQfWYeNSdnp/6g2N2BLseYonQ93GXs5wrobdwANbLsMnwYhCns9ZvB9F25Vtnl0GiOQsrkwy01TEZC9wiAs2WsC8nppi2QTywYl5KVnIXpbWppMYqnWtfU+46pxphwynkmd9yw0CEHT9RP17oeWg6DEP76xLUNe/2pkCDqOu5OLOd8SNLUunAy7vy0+xeczvMlllyy6YFsT0qNl2qUFI3pYsGOzKhFuhvlsmkhNq2oAbV+99k9GiG7ymrvp
    ubuntu@ip-10-15-16-183: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeZUO/YA3/65L1Bqq5Fon3uK8EZQ/snlwfcBu5+ifRAMTh7JiFlfPjKld8tLVflKZi4utl4XnFEzW1OV1M0Kio/HiRipdqJAtCfBK4UPxNrwQTAlENR4RcHQzwNG5/53HsKIWpVyT2HKxKnTZ7x71gjkThQ6YoR6cEKwzwMEAYPimaURJqzybgCiWzTSSQ1LW2WRnTJhLPIFIvPMWT6qW8Chiw60na6Gqt0tv50AoF6Iq49Y7c4uixVzdTCxrwGXFCp+kWewg/YtYRlLRKrEjY8Rlq2F3ED755HIvp7gd8AJ4lI3vxvsqZjzkLNM/nArm/uZ62XSjtq2UDqRCSxVRL
  tasks:
  - install: null
  - clock: null
  - ceph:
      use_existing_cluster: true
  - rbd:
      client.0:
        fs_type: xfs
        image_name: img1
        image_size: 20480
  - workunit:
      clients:
        all:
        - suites/fio_small.sh
  - sequential:
    - reboot:
      - osd.0
    - sleep:
        duration: 300
    - reboot:
      - osd.1
    - sleep:
        duration: 300
2014-10-21T10:58:57.293 INFO:teuthology.run_tasks:Running task internal.save_config...
2014-10-21T10:58:57.294 INFO:teuthology.task.internal:Saving configuration
2014-10-21T10:58:57.299 INFO:teuthology.run_tasks:Running task internal.check_lock...
2014-10-21T10:58:57.299 INFO:teuthology.task.internal:Lock checking disabled.
2014-10-21T10:58:57.299 INFO:teuthology.run_tasks:Running task internal.connect...
2014-10-21T10:58:57.299 INFO:teuthology.task.internal:Opening connections...
2014-10-21T10:58:57.299 DEBUG:teuthology.task.internal:connecting to ubuntu@ip-10-15-16-183
2014-10-21T10:58:59.453 DEBUG:teuthology.task.internal:connecting to ubuntu@ip-10-15-16-182
2014-10-21T10:59:01.581 DEBUG:teuthology.task.internal:connecting to ubuntu@10.15.17.146
2014-10-21T10:59:03.700 INFO:teuthology.task.internal:roles: ubuntu@ip-10-15-16-183 - ['mon.0', 'osd.0', 'osd.1', 'osd.2']
2014-10-21T10:59:03.700 INFO:teuthology.task.internal:roles: ubuntu@ip-10-15-16-182 - ['mon.1', 'osd.3', 'osd.4', 'osd.5']
2014-10-21T10:59:03.700 INFO:teuthology.task.internal:roles: ubuntu@10.15.17.146 - ['mon.2', 'client.0']
2014-10-21T10:59:03.701 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2014-10-21T10:59:03.701 INFO:teuthology.task.internal:Checking for old test directory...
2014-10-21T10:59:03.701 DEBUG:teuthology.orchestra.run:Running [10.15.17.146]: "test '!' -e /home/ubuntu/cephtest" 
2014-10-21T10:59:04.385 DEBUG:teuthology.orchestra.run:Running [10.15.16.182]: "test '!' -e /home/ubuntu/cephtest" 
2014-10-21T10:59:04.969 DEBUG:teuthology.orchestra.run:Running [10.15.16.183]: "test '!' -e /home/ubuntu/cephtest" 
2014-10-21T10:59:05.553 ERROR:teuthology.task.internal:Host ubuntu@10.15.17.146 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2014-10-21T10:59:05.554 ERROR:teuthology.task.internal:Host ubuntu@ip-10-15-16-182 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2014-10-21T10:59:05.558 ERROR:teuthology.task.internal:Host ubuntu@ip-10-15-16-183 has stale test directory /home/ubuntu/cephtest, check lock and cleanup.
2014-10-21T10:59:05.558 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/ems1/Shambhu/Development/storm-1-0-test/qe/teuthology/teuthology/run_tasks.py", line 39, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/ems1/Shambhu/Development/storm-1-0-test/qe/teuthology/teuthology/run_tasks.py", line 27, in run_one_task
    return fn(**kwargs)
  File "/home/ems1/Shambhu/Development/storm-1-0-test/qe/teuthology/teuthology/task/internal.py", line 277, in check_conflict
    raise RuntimeError('Stale jobs detected, aborting.')
RuntimeError: Stale jobs detected, aborting.
2014-10-21T10:59:05.559 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: Stale jobs detected, aborting.
2014-10-21T10:59:05.561 INFO:teuthology.run:Summary data:
{failure_reason: 'Stale jobs detected, aborting.', owner: ems1@ems, success: false}

2014-10-21T10:59:05.561 WARNING:teuthology.report:No job_id found; not reporting results
2014-10-21T10:59:05.561 INFO:teuthology.run:FAIL