Bug #11869: install-deps.sh failed to download missing wheels - Ceph - Ceph

Custom queries

Backports: mimic
Backports: missing release
Backports: nautilus
Bluestore
Bug queue
Bug queue - no subprojects
Bug triage
Ceph backlog
Crash queue
Crash triage
Feature Requests
Feedback
My issues
Need Review
Pending backports
Priority queue
Product Backlog Scrub
Project Triage
Test Failures

Actions

Copy link

Bug #11869

closed

install-deps.sh failed to download missing wheels

Added by Kefu Chai almost 9 years ago. Updated almost 9 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Kefu Chai

Category:

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

if wheelhouse directory exists but not all of the wheels are downloaded, install-deps.sh simply ignores all the requirement.txt in that directory.

when I run ceph make check, it fail at script ceph-detect-init/run-tox.sh.

The logs shows:

Ignoring indexes: https://pypi.python.org/simple
Collecting tox>=1.9
Could not find a version that satisfies the requirement tox>=1.9 (from versions: )
No matching distribution found for tox>=1.9
./ceph-detect-init/run-tox.sh: line 33: tox: command not found

tox is installed in install-deps.sh? But after run install-deps.sh, it return ok even if
tox is not installed successfully. Because in china the network is not so good , some
of the websites we can not ping because of gfw (Greate fire wall).

Could you modify the install-deps.sh, so we chinese user can install tox successfully.

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Kefu Chai almost 9 years ago

so we have two issues here:

1) we should check if we really populate all required wheels in install-deps.sh instead of checking for the wheelhouse directory
2) find a pypi mirror in mainland china. and see if our script can be configured to use it.

I agree. The current script kind of assumes it will exit on error if one of the dependencies fails to be downloaded. But it appears to behave differently.

i think the script did exit on error, but it leaved us with a "wheelhouse" directory not fully populated with required packages. but we took the "wheelhouse" directory as a green light for tox. in Xinze's case, the "wheelhouse" directory was empty: he failed to download any packages.

Actions

Copy link

Updated by Kefu Chai almost 9 years ago

Description updated (diff)

Actions

Copy link

Updated by Xinze Chi almost 9 years ago


(install-deps-python2.7)[root@ceph-code-server ceph-detect-init]# sh run-tox.sh
New python executable in make-check/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
Downloading/unpacking pip>=6.1
  Downloading pip-7.0.3.tar.gz (1.1MB): 736kB downloaded
  Hash of the package https://pypi.python.org/packages/source/p/pip/pip-7.0.3.tar.gz#md5=54cbf5ae000fb3af3367345f5d299d1c (from https://pypi.python.org/simple/pip/) (45f57faaad44593bf30effd2915ac3bd) doesn't match the expected hash 54cbf5ae000fb3af3367345f5d299d1c!
Cleaning up...
Bad md5 hash for package https://pypi.python.org/packages/source/p/pip/pip-7.0.3.tar.gz#md5=54cbf5ae000fb3af3367345f5d299d1c (from https://pypi.python.org/simple/pip/)
Storing complete log in /root/.pip/pip.log
Traceback (most recent call last):
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/bin/pip", line 9, in <module>
    load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/__init__.py", line 148, in main
    return command.main(args[1:], options)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/basecommand.py", line 176, in main
    logger.fatal('Storing complete log in %s' % log_fn)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/log.py", line 48, in fatal
    self.log(self.FATAL, msg, *args, **kw)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/log.py", line 75, in log
    backwardcompat.fwrite(consumer, rendered)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/backwardcompat/__init__.py", line 90, in fwrite
    f.write(s)
ValueError: I/O operation on closed file
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking distribute
Unpacking ./wheelhouse/setuptools-17.0-py2.py3-none-any.whl
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/commands/install.py", line 241, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/req.py", line 1256, in install
    if req.name == 'distribute' and req.installed_version in distribute_req:
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/req.py", line 390, in installed_version
    return self.pkg_info()['version']
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/req.py", line 357, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/req.py", line 293, in egg_info_data
    filename = self.egg_info_path(filename)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/req.py", line 307, in egg_info_path
    filenames = os.listdir(base)
OSError: [Errno 2] No such file or directory: '/home/chixinze/git/ceph/src/ceph-detect-init/make-check/build/distribute/pip-egg-info'

Storing complete log in /root/.pip/pip.log
Traceback (most recent call last):
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/bin/pip", line 9, in <module>
    load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/__init__.py", line 148, in main
    return command.main(args[1:], options)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/basecommand.py", line 176, in main
    logger.fatal('Storing complete log in %s' % log_fn)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/log.py", line 48, in fatal
    self.log(self.FATAL, msg, *args, **kw)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/log.py", line 75, in log
    backwardcompat.fwrite(consumer, rendered)
  File "/home/chixinze/git/ceph/src/ceph-detect-init/make-check/lib/python2.7/site-packages/pip/backwardcompat/__init__.py", line 90, in fwrite
    f.write(s)
ValueError: I/O operation on closed file
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking tox>=1.9
Downloading/unpacking pluggy>=0.3.0,<0.4.0 (from tox>=1.9)
Downloading/unpacking virtualenv>=1.11.2 (from tox>=1.9)
Downloading/unpacking py>=1.4.17 (from tox>=1.9)
Installing collected packages: tox, pluggy, virtualenv, py
Successfully installed tox pluggy virtualenv py
Cleaning up...
run-tox.sh: line 33: tox: command not found

@Kefu Chai as you said

find a pypi mirror in mainland china. and see if our script can be configured to use it"

I think if we can find some deps mirrors in mainland china inclduing pypi mirror, it would be more nice for our chinese developers.

Actions

Copy link

Updated by Kefu Chai almost 9 years ago

Xinze Chi wrote:

[...]

@Kefu Chai as you said

find a pypi mirror in mainland china. and see if our script can be configured to use it"

I think if we can find some deps mirrors in mainland china inclduing pypi mirror, it would be more nice for our chinese developers.

for mirrors for pypi in china, please see http://segmentfault.com/a/1190000000375848#articleHeader2

to configure your pip to use one of them, see https://pip.pypa.io/en/latest/user_guide.html#configuration

for example.

[global]

index-url = https://pypi.mirrors.ustc.edu.cn/simple

Actions

Copy link

Updated by Kefu Chai almost 9 years ago

Status changed from 12 to Resolved

https://github.com/ceph/ceph/pull/4856

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph

Custom queries

Bug #11869

install-deps.sh failed to download missing wheels

Updated by Kefu Chai almost 9 years ago

Updated by Kefu Chai almost 9 years ago

Updated by Xinze Chi almost 9 years ago

Updated by Kefu Chai almost 9 years ago

Updated by Kefu Chai almost 9 years ago