Bug #16169
closedceph-detect-init target fails for Jewel build
0%
Description
In case that log output disappears:
Installing setuptools, pkg_resources, pip, wheel... CXX kv/libkv_a-KeyValueDB.o
CXX kv/libkv_a-LevelDBStore.o
CXX kv/libkv_a-RocksDBStore.o
CXX mon/Monitor.o
Complete output from command /tmp/ceph-detect-ini...ualenv/bin/python2.7 - setuptools pkg_resources pip wheel:
Collecting setuptools
Exception:
Traceback (most recent call last):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 328, in run
wb.build(autobuilding=True)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/wheel.py", line 748, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 440, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 398, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 543, in _get_pages
page = self._get_page(location)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 646, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 755, in get_page
"Cache-Control": "max-age=600",
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send
timeout=timeout
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
----------------------------------------
...Installing setuptools, pkg_resources, pip, wheel...done.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2363, in <module>
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 719, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 988, in create_environment
download=download,
File "/usr/lib/python3/dist-packages/virtualenv.py", line 918, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 812, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /tmp/ceph-detect-ini...ualenv/bin/python2.7 - setuptools pkg_resources pip wheel failed with error code 2
Updated by Alfredo Deza almost 8 years ago
Updated by Alfredo Deza over 7 years ago
- Assignee set to Loïc Dachary
- Priority changed from Normal to Immediate
Hi Loic, this is still an issue and we have narrowed down to being a problem for building Jewel in a Xenial box.
No other branches have this problem and we are now trying to cut 10.2.3 and facing the exact same problem
See for example: https://jenkins.ceph.com/job/ceph-dev-build/649/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
cd ./ceph-detect-init ; ../tools/setup-virtualenv.sh /tmp/ceph-detect-init-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-detect-init-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e .
New python executable in /tmp/ceph-detect-init-virtualenv/bin/python2.7
Also creating executable in /tmp/ceph-detect-init-virtualenv/bin/python
Installing setuptools, pkg_resources, pip, wheel...cd ./ceph-disk ; ../tools/setup-virtualenv.sh /tmp/ceph-disk-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-disk-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e .
New python executable in /tmp/ceph-disk-virtualenv/bin/python2.7
Also creating executable in /tmp/ceph-disk-virtualenv/bin/python
Installing setuptools, pkg_resources, pip, wheel...
Traceback (most recent call last):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 317, in run
requirement_set.prepare_files(finder)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file
finder, self.upgrade, require_hashes)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 440, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 398, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 543, in _get_pages
page = self._get_page(location)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 646, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 755, in get_page
"Cache-Control": "max-age=600",
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send
timeout=timeout
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 609, in urlopen
_stacktrace=sys.exc_info()[2])
File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment
total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
----------------------------------------
I think that part of the error is that it is using Python's PIP from /usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl
Like I mentioned, this only happens in Jewel. Could you please take some time to investigate this? Maybe something needs to get backported/changed for the ceph-disk virtualenv setup for Jewel?
Updated by Loïc Dachary over 7 years ago
- Assignee changed from Loïc Dachary to Kefu Chai
@Kefu Chai would you mind taking a look ? There has been two dozen changes since I last modified this part of the code about six months ago. Given the urgency of the fix, you are probably better equipped than I am. Thanks :-)
Updated by Dan Mick over 7 years ago
It seems like trying to mix /usr/share/python-wheels and a local wheelhouse is doomed to failure. The /usr/share version of requests does not include a packages directory with local packages, so gets the system urllib3 and chardet. I don't understand why the local wheelhouse is there, because the code doesn't have comments explaining it.
Updated by Nathan Cutler over 7 years ago
I seem to recall that the local wheelhouse is there to accommodate building in environments (like OBS) that aren't connected to the Internet.
Updated by Dan Mick over 7 years ago
I can't figure out what in the jenkins build is doing the same job as install-deps.sh does, to prepopulate wheelhouse. I don't yet understand how that should affect the installation of pip in the virtualenv, but certainly the code that tries to install wheels from wheelhouse is not working.
Updated by Loïc Dachary over 7 years ago
I think this may be another case of jenkins not cleaning after itself and starting a job that is under the influence of leftovers from a previous job. Or that jenkins is indeed installing pip from package and not providing the job with a pristine environment. There are two ways out of this: fixing jenkins or backporting one of the workarounds implemented in Ceph to compensate for jenkins shortcomings. I'm unfortunately not familiar enough with those workarounds to backport them quickly :-(
Updated by Alfredo Deza over 7 years ago
@Loïc Dachary I don't think this is a case of environmental leftovers from other builds. We can consistently replicate this problem for the Jewel branch of Ceph and in Xenial only.
Every other branch in Ceph for all the other combination of supported distribution passes. This is specific to just Jewel and Xenial.
Updated by Kefu Chai over 7 years ago
so we are using pbuilder to build the package, and pbuilder is basically a clean room created by chroot:
- [1] in kraken, we don't create venv in "make all", so the only venv is created by ansible, so it has access to network.
Ubuntu is unknown, dependencies will have to be installed manually. + pkgs=("chacractl>=0.0.4") + install_python_packages 'pkgs[@]' + virtualenv /tmp/venv.UIQ6VEp17V New python executable in /tmp/venv.UIQ6VEp17V/bin/python2 Also creating executable in /tmp/venv.UIQ6VEp17V/bin/python Installing setuptools, pkg_resources, pip, wheel...done. - [2] in jewel, we create venv in "make all" (see [3]) for preparing "make check", but
virtualenvdoes not have access to network at that moment, probably that's why we are in this dilemma.
and pbuilder have no access to its host environment, that's why "BUILD_HOME" does not work.
The directory '/home/jenkins-build/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
we can either
- fulfill the requirement of virtualenv using
BINDMOUNTS, tomount --binda host directory in the chroot env, a good candidates is "/home/jenkins-build/.cache/", but we might need to pre-populate this directory with required wheels for better performance. another candidate is a ccache directory shared among builds! - or, pre-populate the wheels in the wheelhouse directory as expected by setup-virtualenv.sh script. and put them into the dist tarball. so the build task can use them without the need to access network.
- or, enable the network access from pbuilder, to put "USENETWORK=yes" in
~/.pbuilderrc. - or, add yet another workaround in setup-virtualenv.sh to tell if it is in a pbuild env, if it is, then do nothing and return. how? maybe we can check if
$HOMEis readable?
personally, i'd prefer option 2.
---
[1] success xenial build of ceph-11.0.0: https://jenkins.ceph.com/job/ceph-dev-build/655/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[2] failed xenial build of ceph-10.2.2: https://jenkins.ceph.com/job/ceph-dev-build/649/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[3]
$ git grep -w ceph-disk-all
src/ceph-disk/Makefile.am:ceph-disk-all: ${CEPH_DISK_VIRTUALENV}
src/ceph-disk/Makefile.am:LOCAL_ALL += ceph-disk-all
Updated by Alfredo Deza over 7 years ago
The " parent directory is not owned by the current user and the cache has been disabled" error is misleading. Setting the BUILD_HOME variable to `mktemp -d` gets rid of the error (see [0])
In this case, $HOME becomes readable and the error goes away. But the traceback persists because it is using the cached wheels directory provided by Xenial which has the
buggy version of pip that has that error (8.1.1)
Updated by Kefu Chai over 7 years ago
- Assignee changed from Kefu Chai to Alfredo Deza
Updated by Alfredo Deza over 7 years ago
- Status changed from New to Resolved
We are now having to allow network access when building. This sounds very unreasonable and it isn't something that happened before.
Pull Request to work around this issue in build scripts: https://github.com/ceph/ceph-build/pull/486
This issue would help prevent this in the future: #15260