Ceph

ubuntu@ceph-admin:~/teuthology$ sudo ./virtualenv/bin/teuthology examples/3node_rgw.yaml
2015-07-22 16:43:21,624.624 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-22 16:43:21,629.629 INFO:teuthology.run:Tasks not found; will attempt to fetch
2015-07-22 16:43:21,630.630 INFO:teuthology.repo_utils:Fetching from upstream into /home/ubuntu/src/ceph-qa-suite_master
2015-07-22 16:43:39,525.525 INFO:teuthology.repo_utils:Resetting repo at /home/ubuntu/src/ceph-qa-suite_master to branch master
2015-07-22 16:43:39,613.613 INFO:teuthology.run_tasks:Running task internal.check_packages...
2015-07-22 16:43:39,613.613 INFO:teuthology.task.internal:Checking packages...
2015-07-22 16:43:39,614.614 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash 'None'
2015-07-22 16:43:39,614.614 INFO:teuthology.run_tasks:Running task internal.save_config...
2015-07-22 16:43:39,615.615 INFO:teuthology.task.internal:Saving configuration
2015-07-22 16:43:39,615.615 INFO:teuthology.run_tasks:Running task internal.check_lock...
2015-07-22 16:43:39,615.615 INFO:teuthology.task.internal:Lock checking disabled.
2015-07-22 16:43:39,616.616 INFO:teuthology.run_tasks:Running task internal.add_remotes...
2015-07-22 16:43:39,618.618 INFO:teuthology.task.internal:roles: ubuntu@ceph-node1.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0']
2015-07-22 16:43:39,618.618 INFO:teuthology.task.internal:roles: ubuntu@ceph-node3.front.sepia.ceph.com - ['mon.b', 'mds.a', 'osd.1']
2015-07-22 16:43:39,618.618 INFO:teuthology.task.internal:roles: ubuntu@ceph-node2.front.sepia.ceph.com - ['client.0']
2015-07-22 16:43:39,619.619 INFO:teuthology.run_tasks:Running task internal.connect...
2015-07-22 16:43:39,619.619 INFO:teuthology.task.internal:Opening connections...
2015-07-22 16:43:39,621.621 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'ceph-node1.front.sepia.ceph.com', 'timeout': 60}
2015-07-22 16:43:40,220.220 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'ceph-node3.front.sepia.ceph.com', 'timeout': 60}
2015-07-22 16:43:40,842.842 INFO:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'ceph-node2.front.sepia.ceph.com', 'timeout': 60}
2015-07-22 16:43:41,440.440 INFO:teuthology.run_tasks:Running task internal.push_inventory...
2015-07-22 16:43:41,440.440 INFO:teuthology.orchestra.run.ceph-node1:Running: 'uname m'
2015-07-22 16:43:41,514.514 INFO:teuthology.orchestra.run.ceph-node1:Running: "python -c 'import platform; print platform.linux_distribution()'"
2015-07-22 16:43:41,811.811 INFO:teuthology.lock:Updating ceph-node1.front.sepia.ceph.com on lock server
2015-07-22 16:44:01,847.847 ERROR:teuthology.task.internal:Error pushing inventory
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 335, in push_inventory
push()
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 333, in push
lock.update_inventory(info)
File "/home/ubuntu/teuthology/teuthology/lock.py", line 636, in update_inventory
headers={'content-type': 'application/json'},
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 122, in put
return request('put', url, data=data, **kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
2015-07-22 16:44:01,849.849 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles...
2015-07-22 16:44:01,849.849 INFO:teuthology.run_tasks:Running task internal.check_conflict...
2015-07-22 16:44:01,850.850 INFO:teuthology.task.internal:Checking for old test directory...
2015-07-22 16:44:01,850.850 INFO:teuthology.orchestra.run.ceph-node1:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-22 16:44:01,857.857 INFO:teuthology.orchestra.run.ceph-node2:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-22 16:44:01,895.895 INFO:teuthology.orchestra.run.ceph-node3:Running: "test '!' -e /home/ubuntu/cephtest"
2015-07-22 16:44:01,906.906 INFO:teuthology.run_tasks:Running task internal.check_ceph_data...
2015-07-22 16:44:01,906.906 INFO:teuthology.task.internal:Checking for old /var/lib/ceph...
2015-07-22 16:44:01,906.906 INFO:teuthology.orchestra.run.ceph-node1:Running: "test '!' -e /var/lib/ceph"
2015-07-22 16:44:01,910.910 INFO:teuthology.orchestra.run.ceph-node2:Running: "test '!' -e /var/lib/ceph"
2015-07-22 16:44:01,947.947 INFO:teuthology.orchestra.run.ceph-node3:Running: "test '!' -e /var/lib/ceph"
2015-07-22 16:44:01,958.958 INFO:teuthology.run_tasks:Running task internal.vm_setup...
2015-07-22 16:44:01,958.958 INFO:teuthology.run_tasks:Running task internal.base...
2015-07-22 16:44:01,958.958 INFO:teuthology.task.internal:Creating test directory...
2015-07-22 16:44:01,959.959 INFO:teuthology.orchestra.run.ceph-node1:Running: 'mkdir -p -m0755 - /home/ubuntu/cephtest'
2015-07-22 16:44:01,962.962 INFO:teuthology.orchestra.run.ceph-node2:Running: 'mkdir p -m0755 - /home/ubuntu/cephtest'
2015-07-22 16:44:02,001.001 INFO:teuthology.orchestra.run.ceph-node3:Running: 'mkdir p -m0755 - /home/ubuntu/cephtest'
2015-07-22 16:44:02,062.062 INFO:teuthology.run_tasks:Running task internal.archive...
2015-07-22 16:44:02,063.063 INFO:teuthology.task.internal:Creating archive directory...
2015-07-22 16:44:02,064.064 INFO:teuthology.orchestra.run.ceph-node1:Running: 'install d -m0755 - /home/ubuntu/cephtest/archive'
2015-07-22 16:44:02,068.068 INFO:teuthology.orchestra.run.ceph-node2:Running: 'install d -m0755 - /home/ubuntu/cephtest/archive'
2015-07-22 16:44:02,072.072 INFO:teuthology.orchestra.run.ceph-node3:Running: 'install d -m0755 - /home/ubuntu/cephtest/archive'
2015-07-22 16:44:02,134.134 INFO:teuthology.run_tasks:Running task internal.coredump...
2015-07-22 16:44:02,135.135 INFO:teuthology.task.internal:Enabling coredump saving...
2015-07-22 16:44:02,136.136 INFO:teuthology.orchestra.run.ceph-node1:Running: 'install d -m0755 - /home/ubuntu/cephtest/archive/coredump && sudo sysctl w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core'
2015-07-22 16:44:02,141.141 INFO:teuthology.orchestra.run.ceph-node2:Running: 'install -d -m0755 - /home/ubuntu/cephtest/archive/coredump && sudo sysctl w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core'
2015-07-22 16:44:02,145.145 INFO:teuthology.orchestra.run.ceph-node3:Running: 'install -d -m0755 - /home/ubuntu/cephtest/archive/coredump && sudo sysctl w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core'
2015-07-22 16:44:02,155.155 INFO:teuthology.orchestra.run.ceph-node2.stderr:sudo: no tty present and no askpass program specified
2015-07-22 16:44:02,326.326 INFO:teuthology.orchestra.run.ceph-node1.stderr:sudo: no tty present and no askpass program specified
2015-07-22 16:44:02,329.329 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 56, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/home/ubuntu/teuthology/teuthology/task/internal.py", line 532, in coredump
wait=False,
File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 401, in wait
proc.wait()
File "/home/ubuntu/teuthology/teuthology/orchestra/run.py", line 114, in wait
label=self.label)
CommandFailedError: Command failed on ceph-node1 with status 1: 'install -d -m0755 - /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core'
2015-07-22 16:44:02,337.337 WARNING:teuthology.run_tasks:Saw failure during task execution, going into interactive mode...
Ceph test interactive mode, use ctx to interact with the cluster, press control-D to exit...

2015-07-22 16:46:50,204.204 INFO:teuthology.task.internal:Removing archive directory...
2015-07-22 16:46:50,205.205 INFO:teuthology.orchestra.run.ceph-node1:Running: 'rm rf - /home/ubuntu/cephtest/archive'
2015-07-22 16:46:50,211.211 INFO:teuthology.orchestra.run.ceph-node3.stderr:sudo: no tty present and no askpass program specified
2015-07-22 16:46:50,214.214 INFO:teuthology.run:Summary data: {failure_reason: 'Command failed on ceph-node1 with status 1: ''install d -m0755
- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core''',
owner: root@ceph-admin, status: fail, success: false}

2015-07-22 16:46:50,215.215 WARNING:teuthology.report:No job_id found; not reporting results
2015-07-22 16:46:50,215.215 INFO:teuthology.run:FAIL
2015-07-22 16:46:50,215.215 INFO:teuthology.task.internal:Tidying up after the test...
2015-07-22 16:46:50,216.216 INFO:teuthology.orchestra.run.ceph-node1:Running: 'rmdir -- /home/ubuntu/cephtest'
2015-07-22 16:46:50,286.286 INFO:teuthology.orchestra.run.ceph-node2:Running: 'rmdir -- /home/ubuntu/cephtest'
2015-07-22 16:46:50,293.293 INFO:teuthology.orchestra.run.ceph-node3:Running: 'rmdir -- /home/ubuntu/cephtest'
2015-07-22 16:46:50,315.315 INFO:teuthology.orchestra.run.ceph-node2.stderr:rmdir: failed to remove ‘/home/ubuntu/cephtest’: Directory not empty
2015-07-22 16:46:50,324.324 INFO:teuthology.orchestra.run.ceph-node1.stderr:rmdir: failed to remove ‘/home/ubuntu/cephtest’: Directory not empty
Exception teuthology.exceptions.CommandFailedError: CommandFailedError() in <generator object base at 0x7fe8dda09550> ignored