Bug #10330
apt-mirror timeouts
0%
Description
2014-12-15T20:05:04.448 INFO:teuthology.orchestra.run.plana37.stdout:Cleaning up... 2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:Exception: 2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:Traceback (most recent call last): 2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main 2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout: status = self.run(options, args) 2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: do_download, 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: self.session, 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/download.py", line 546, in unpack_http_url 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: resp = session.get(target_url, stream=True) 2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 468, in get 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: return self.request('GET', url, **kwargs) 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/download.py", line 237, in request 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: return super(PipSession, self).request(method, url, *args, **kwargs) 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 456, in request 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: resp = self.send(prep, **send_kwargs) 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 559, in send 2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout: r = adapter.send(request, **kwargs) 2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout: File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 384, in send 2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout: raise Timeout(e, request=request) 2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:Timeout: HTTPConnectionPool(host='apt-mirror.front.sepia.ceph.com', port=80): Read timed out. (read timeout=15) 2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout: 2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:Storing debug log for failure in /home/ubuntu/.pip/pip.log 2014-12-15T20:05:04.561 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure './bootstrap', File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run r.wait() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap' 2014-12-15T20:05:04.564 INFO:teuthology.orchestra.run.plana37:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.1 user rm --uid foo.client.1 --purge-data' 2014-12-15T20:05:04.922 INFO:teuthology.orchestra.run.plana37:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.1 user rm --uid bar.client.1 --purge-data' 2014-12-15T20:05:05.248 INFO:teuthology.task.swift:Removing swift... 2014-12-15T20:05:05.249 INFO:teuthology.orchestra.run.plana37:Running: 'rm -rf /home/ubuntu/cephtest/swift' 2014-12-15T20:05:05.284 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 256, in task lambda: run_tests(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure './bootstrap', File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run r.wait() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap' 2014-12-15T20:05:05.284 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2014-12-15T20:05:05.284 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested yield vars File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/rgw.py", line 846, in task yield File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 256, in task lambda: run_tests(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure './bootstrap', File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run r.wait() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap'
Related issues
History
#1 Updated by Zack Cerza over 9 years ago
- Project changed from teuthology to devops
- Subject changed from "Storing debug log for failure" in upgrade:firefly-x-next-distro-basic-multi run to apt-mirror timeouts
- Assignee set to Sandon Van Ness
- Priority changed from Normal to High
#4 Updated by Yuri Weinstein over 9 years ago
Same issue in run http://pulpito.ceph.com/teuthology-2014-12-21_19:13:02-upgrade:dumpling-x-firefly-distro-basic-vps/
Jobs ['671769', '671775', '671787']
#5 Updated by Yuri Weinstein about 9 years ago
Similar in
Run: http://pulpito.ceph.com/teuthology-2015-01-22_14:00:18-upgrade:firefly-x-giant-distro-basic-vps/
Job: ['718109']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-22_14:00:18-upgrade:firefly-x-giant-distro-basic-vps/718109/
2015-01-22T18:54:40.418 INFO:teuthology.orchestra.run.vpm181.stdout:Downloading/unpacking pip from http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa 2015-01-22T18:54:40.418 INFO:teuthology.orchestra.run.vpm181.stdout: Error timed out while getting http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa
#6 Updated by Yuri Weinstein about 9 years ago
More like these.
Run: http://pulpito.ceph.com/teuthology-2015-01-27_17:05:04-upgrade:giant-x-next-distro-basic-multi/
Job: [726022]
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-27_17:05:04-upgrade:giant-x-next-distro-basic-multi/726022/
2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout:Downloading/unpacking pip from http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa 2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout: Error timed out while getting http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa 2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout:Exception:
#7 Updated by Zack Cerza about 9 years ago
I have a caching nginx proxy up on mira056 that appears to be working well.
#8 Updated by Zack Cerza about 9 years ago
- Status changed from New to 7
- Assignee changed from Sandon Van Ness to Yuri Weinstein
We think this is fixed. Yuri will resolve this if he doesn't see it again soon.
#9 Updated by Yuri Weinstein about 9 years ago
- Status changed from 7 to Resolved
Looks fixed for now