Project

General

Profile

Bug #10330

apt-mirror timeouts

Added by Yuri Weinstein about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
High
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-12-15_17:13:01-upgrade:firefly-x-next-distro-basic-multi/661173/

2014-12-15T20:05:04.448 INFO:teuthology.orchestra.run.plana37.stdout:Cleaning up...
2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:Exception:
2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:Traceback (most recent call last):
2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:    status = self.run(options, args)
2014-12-15T20:05:04.535 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/commands/install.py", line 278, in run
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/req.py", line 1197, in prepare_files
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:    do_download,
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/req.py", line 1375, in unpack_url
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:    self.session,
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/download.py", line 546, in unpack_http_url
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:    resp = session.get(target_url, stream=True)
2014-12-15T20:05:04.536 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 468, in get
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:    return self.request('GET', url, **kwargs)
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/download.py", line 237, in request
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:    return super(PipSession, self).request(method, url, *args, **kwargs)
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 456, in request
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:    resp = self.send(prep, **send_kwargs)
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/sessions.py", line 559, in send
2014-12-15T20:05:04.537 INFO:teuthology.orchestra.run.plana37.stdout:    r = adapter.send(request, **kwargs)
2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:  File "/home/ubuntu/cephtest/swift/virtualenv/local/lib/python2.7/site-packages/pip/_vendor/requests/adapters.py", line 384, in send
2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:    raise Timeout(e, request=request)
2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:Timeout: HTTPConnectionPool(host='apt-mirror.front.sepia.ceph.com', port=80): Read timed out. (read timeout=15)
2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:
2014-12-15T20:05:04.538 INFO:teuthology.orchestra.run.plana37.stdout:Storing debug log for failure in /home/ubuntu/.pip/pip.log
2014-12-15T20:05:04.561 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure
    './bootstrap',
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap'
2014-12-15T20:05:04.564 INFO:teuthology.orchestra.run.plana37:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.1 user rm --uid foo.client.1 --purge-data'
2014-12-15T20:05:04.922 INFO:teuthology.orchestra.run.plana37:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.1 user rm --uid bar.client.1 --purge-data'
2014-12-15T20:05:05.248 INFO:teuthology.task.swift:Removing swift...
2014-12-15T20:05:05.249 INFO:teuthology.orchestra.run.plana37:Running: 'rm -rf /home/ubuntu/cephtest/swift'
2014-12-15T20:05:05.284 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 256, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure
    './bootstrap',
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap'
2014-12-15T20:05:05.284 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2014-12-15T20:05:05.284 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested
    yield vars
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/rgw.py", line 846, in task
    yield
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 256, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 143, in configure
    './bootstrap',
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 368, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 106, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana37 with status 2: 'cd /home/ubuntu/cephtest/swift && ./bootstrap'

Related issues

Related to devops - Bug #10575: "Error timed out while getting http://apt-mirror.front.sepia.ceph.com" Won't Fix 01/19/2015

History

#1 Updated by Zack Cerza about 9 years ago

  • Project changed from teuthology to devops
  • Subject changed from "Storing debug log for failure" in upgrade:firefly-x-next-distro-basic-multi run to apt-mirror timeouts
  • Assignee set to Sandon Van Ness
  • Priority changed from Normal to High

#5 Updated by Yuri Weinstein about 9 years ago

Similar in
Run: http://pulpito.ceph.com/teuthology-2015-01-22_14:00:18-upgrade:firefly-x-giant-distro-basic-vps/
Job: ['718109']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-22_14:00:18-upgrade:firefly-x-giant-distro-basic-vps/718109/

2015-01-22T18:54:40.418 INFO:teuthology.orchestra.run.vpm181.stdout:Downloading/unpacking pip from http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa
2015-01-22T18:54:40.418 INFO:teuthology.orchestra.run.vpm181.stdout:  Error timed out while getting http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa

#6 Updated by Yuri Weinstein about 9 years ago

More like these.
Run: http://pulpito.ceph.com/teuthology-2015-01-27_17:05:04-upgrade:giant-x-next-distro-basic-multi/
Job: [726022]
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-27_17:05:04-upgrade:giant-x-next-distro-basic-multi/726022/

2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout:Downloading/unpacking pip from http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa
2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout:  Error timed out while getting http://apt-mirror.front.sepia.ceph.com/pypi/packages/source/p/pip/pip-6.0.6.tar.gz#md5=bbb17814bdf82187f46aaf9cec6b6caa
2015-01-28T08:27:30.189 INFO:teuthology.orchestra.run.plana34.stdout:Exception:

#7 Updated by Zack Cerza about 9 years ago

I have a caching nginx proxy up on mira056 that appears to be working well.

#8 Updated by Zack Cerza about 9 years ago

  • Status changed from New to 7
  • Assignee changed from Sandon Van Ness to Yuri Weinstein

We think this is fixed. Yuri will resolve this if he doesn't see it again soon.

#9 Updated by Yuri Weinstein about 9 years ago

  • Status changed from 7 to Resolved

Looks fixed for now

Also available in: Atom PDF