Ceph

Run http://pulpito.ceph.com/teuthology-2016-05-26_18:15:02-upgrade:hammer-x-jewel-distro-basic-vps/
Jobs: 217761, 217771
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-05-26_18:15:02-upgrade:hammer-x-jewel-distro-basic-vps/217761/teuthology.log

2016-05-26T23:07:13.691 INFO:teuthology.orchestra.run.vpm136:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph rmpool unique_pool_13 unique_pool_13 --yes-i-really-really-mean-it'
2016-05-26T23:07:16.734 INFO:teuthology.orchestra.run.vpm136.stderr:2016-05-27 06:07:16.735874 7f0497d03a40 -1 asok(0x7f04997dfc60) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.admin.28512.asok': (13) Permission denied
2016-05-26T23:07:17.164 INFO:teuthology.orchestra.run.vpm136.stdout:successfully deleted pool unique_pool_13
2016-05-26T23:07:17.167 DEBUG:teuthology.run_tasks:Unwinding manager swift
2016-05-26T23:07:17.213 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2016-05-26T23:07:17.271 INFO:teuthology.misc:Shutting down rgw daemons...
2016-05-26T23:07:17.271 DEBUG:tasks.rgw.client.0:waiting for process to exit
2016-05-26T23:07:17.272 INFO:teuthology.orchestra.run:waiting for 300
2016-05-26T23:07:17.274 INFO:tasks.rgw.client.0.vpm035.stdout:2016-05-27 06:07:17.274957 7f95f574e880 -1 shutting down
2016-05-26T23:07:23.273 INFO:tasks.rgw.client.0:Stopped
2016-05-26T23:07:23.274 INFO:teuthology.orchestra.run.vpm035:Running: 'rm -f /home/ubuntu/cephtest/rgw.opslog.client.0.sock'
2016-05-26T23:07:23.336 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart
2016-05-26T23:07:23.394 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart
2016-05-26T23:07:23.447 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart
2016-05-26T23:07:23.494 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds
2016-05-26T23:07:23.547 INFO:tasks.thrashosds:joining thrashosds
2016-05-26T23:07:23.547 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 139, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/thrashosds.py", line 184, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 431, in do_join
    self.thread.get()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 454, in get
    self._raise_exception()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 171, in _raise_exception
    reraise(*self.exc_info)
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 646, in wrapper
    return func(self)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 693, in do_thrash
    timeout=self.config.get('timeout')
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 1673, in wait_for_recovery
    'failed to recover before timeout expired'
AssertionError: failed to recover before timeout expired
2016-05-26T23:07:23.548 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart
2016-05-26T23:07:23.600 DEBUG:teuthology.run_tasks:Unwinding manager install.upgrade
2016-05-26T23:07:23.654 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2016-05-26T23:07:23.707 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph pg dump --format=json'
2016-05-26T23:07:25.413 INFO:teuthology.orchestra.run.vpm136.stderr:dumped all in format json
2016-05-26T23:07:26.444 INFO:tasks.ceph:Scrubbing osd.3
2016-05-26T23:07:26.444 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 3'
2016-05-26T23:07:26.767 INFO:teuthology.orchestra.run.vpm136.stderr:osd.3 instructed to deep-scrub
2016-05-26T23:07:26.779 INFO:tasks.ceph:Scrubbing osd.4
2016-05-26T23:07:26.779 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 4'
2016-05-26T23:07:27.104 INFO:teuthology.orchestra.run.vpm136.stderr:osd.4 instructed to deep-scrub
2016-05-26T23:07:27.116 INFO:tasks.ceph:Scrubbing osd.5
2016-05-26T23:07:27.117 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 5'
2016-05-26T23:07:27.437 INFO:teuthology.orchestra.run.vpm136.stderr:osd.5 instructed to deep-scrub
2016-05-26T23:07:27.448 INFO:tasks.ceph:Scrubbing osd.0
2016-05-26T23:07:27.448 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 0'
2016-05-26T23:07:27.776 INFO:teuthology.orchestra.run.vpm136.stderr:osd.0 instructed to deep-scrub
2016-05-26T23:07:27.788 INFO:tasks.ceph:Scrubbing osd.1
2016-05-26T23:07:27.788 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 1'
2016-05-26T23:07:28.121 INFO:teuthology.orchestra.run.vpm136.stderr:osd.1 instructed to deep-scrub
2016-05-26T23:07:28.134 INFO:tasks.ceph:Scrubbing osd.2
2016-05-26T23:07:28.134 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd deep-scrub 2'
2016-05-26T23:07:28.463 INFO:teuthology.orchestra.run.vpm136.stderr:osd.2 instructed to deep-scrub
2016-05-26T23:07:28.477 INFO:teuthology.orchestra.run.vpm136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph pg dump --format=json'
2016-05-26T23:07:28.817 INFO:teuthology.orchestra.run.vpm136.stderr:dumped all in format json
2016-05-26T23:07:28.876 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph.py", line 1494, in task
    yield
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 139, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/thrashosds.py", line 184, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 431, in do_join
    self.thread.get()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 454, in get
    self._raise_exception()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 171, in _raise_exception
    reraise(*self.exc_info)
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 646, in wrapper
    return func(self)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 693, in do_thrash
    timeout=self.config.get('timeout')
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_manager.py", line 1673, in wait_for_recovery
    'failed to recover before timeout expired'
AssertionError: failed to recover before timeout expired