Bug #16169
closedceph-detect-init target fails for Jewel build
0%
Description
In case that log output disappears:
Installing setuptools, pkg_resources, pip, wheel... CXX kv/libkv_a-KeyValueDB.o CXX kv/libkv_a-LevelDBStore.o CXX kv/libkv_a-RocksDBStore.o CXX mon/Monitor.o Complete output from command /tmp/ceph-detect-ini...ualenv/bin/python2.7 - setuptools pkg_resources pip wheel: Collecting setuptools Exception: Traceback (most recent call last): File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main status = self.run(options, args) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 328, in run wb.build(autobuilding=True) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/wheel.py", line 748, in build self.requirement_set.prepare_files(self.finder) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file finder, self.upgrade, require_hashes) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 440, in find_requirement all_candidates = self.find_all_candidates(req.name) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 398, in find_all_candidates for page in self._get_pages(url_locations, project_name): File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 543, in _get_pages page = self._get_page(location) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 646, in _get_page return HTMLPage.get_page(link, session=self.session) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 755, in get_page "Cache-Control": "max-age=600", File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get return self.request('GET', url, **kwargs) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request return super(PipSession, self).request(method, url, *args, **kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send resp = super(CacheControlAdapter, self).send(request, **kw) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send timeout=timeout File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 609, in urlopen _stacktrace=sys.exc_info()[2]) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment total -= 1 TypeError: unsupported operand type(s) for -=: 'Retry' and 'int' ---------------------------------------- ...Installing setuptools, pkg_resources, pip, wheel...done. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/virtualenv.py", line 2363, in <module> main() File "/usr/lib/python3/dist-packages/virtualenv.py", line 719, in main symlink=options.symlink) File "/usr/lib/python3/dist-packages/virtualenv.py", line 988, in create_environment download=download, File "/usr/lib/python3/dist-packages/virtualenv.py", line 918, in install_wheel call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT) File "/usr/lib/python3/dist-packages/virtualenv.py", line 812, in call_subprocess % (cmd_desc, proc.returncode)) OSError: Command /tmp/ceph-detect-ini...ualenv/bin/python2.7 - setuptools pkg_resources pip wheel failed with error code 2
Updated by Alfredo Deza almost 8 years ago
Updated by Alfredo Deza over 7 years ago
- Assignee set to Loïc Dachary
- Priority changed from Normal to Immediate
Hi Loic, this is still an issue and we have narrowed down to being a problem for building Jewel in a Xenial box.
No other branches have this problem and we are now trying to cut 10.2.3 and facing the exact same problem
See for example: https://jenkins.ceph.com/job/ceph-dev-build/649/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
cd ./ceph-detect-init ; ../tools/setup-virtualenv.sh /tmp/ceph-detect-init-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-detect-init-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e . New python executable in /tmp/ceph-detect-init-virtualenv/bin/python2.7 Also creating executable in /tmp/ceph-detect-init-virtualenv/bin/python Installing setuptools, pkg_resources, pip, wheel...cd ./ceph-disk ; ../tools/setup-virtualenv.sh /tmp/ceph-disk-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-disk-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e . New python executable in /tmp/ceph-disk-virtualenv/bin/python2.7 Also creating executable in /tmp/ceph-disk-virtualenv/bin/python Installing setuptools, pkg_resources, pip, wheel... Traceback (most recent call last): File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main status = self.run(options, args) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 317, in run requirement_set.prepare_files(finder) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file finder, self.upgrade, require_hashes) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 440, in find_requirement all_candidates = self.find_all_candidates(req.name) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 398, in find_all_candidates for page in self._get_pages(url_locations, project_name): File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 543, in _get_pages page = self._get_page(location) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 646, in _get_page return HTMLPage.get_page(link, session=self.session) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 755, in get_page "Cache-Control": "max-age=600", File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get return self.request('GET', url, **kwargs) File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request return super(PipSession, self).request(method, url, *args, **kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send resp = super(CacheControlAdapter, self).send(request, **kw) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send timeout=timeout File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 609, in urlopen _stacktrace=sys.exc_info()[2]) File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment total -= 1 TypeError: unsupported operand type(s) for -=: 'Retry' and 'int' ----------------------------------------
I think that part of the error is that it is using Python's PIP from /usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl
Like I mentioned, this only happens in Jewel. Could you please take some time to investigate this? Maybe something needs to get backported/changed for the ceph-disk virtualenv setup for Jewel?
Updated by Loïc Dachary over 7 years ago
- Assignee changed from Loïc Dachary to Kefu Chai
@Kefu Chai would you mind taking a look ? There has been two dozen changes since I last modified this part of the code about six months ago. Given the urgency of the fix, you are probably better equipped than I am. Thanks :-)
Updated by Dan Mick over 7 years ago
It seems like trying to mix /usr/share/python-wheels and a local wheelhouse is doomed to failure. The /usr/share version of requests does not include a packages directory with local packages, so gets the system urllib3 and chardet. I don't understand why the local wheelhouse is there, because the code doesn't have comments explaining it.
Updated by Nathan Cutler over 7 years ago
I seem to recall that the local wheelhouse is there to accommodate building in environments (like OBS) that aren't connected to the Internet.
Updated by Dan Mick over 7 years ago
I can't figure out what in the jenkins build is doing the same job as install-deps.sh does, to prepopulate wheelhouse. I don't yet understand how that should affect the installation of pip in the virtualenv, but certainly the code that tries to install wheels from wheelhouse is not working.
Updated by Loïc Dachary over 7 years ago
I think this may be another case of jenkins not cleaning after itself and starting a job that is under the influence of leftovers from a previous job. Or that jenkins is indeed installing pip from package and not providing the job with a pristine environment. There are two ways out of this: fixing jenkins or backporting one of the workarounds implemented in Ceph to compensate for jenkins shortcomings. I'm unfortunately not familiar enough with those workarounds to backport them quickly :-(
Updated by Alfredo Deza over 7 years ago
@Loïc Dachary I don't think this is a case of environmental leftovers from other builds. We can consistently replicate this problem for the Jewel branch of Ceph and in Xenial only.
Every other branch in Ceph for all the other combination of supported distribution passes. This is specific to just Jewel and Xenial.
Updated by Kefu Chai over 7 years ago
so we are using pbuilder to build the package, and pbuilder is basically a clean room created by chroot:
- [1] in kraken, we don't create venv in "make all", so the only venv is created by ansible, so it has access to network.
Ubuntu is unknown, dependencies will have to be installed manually. + pkgs=("chacractl>=0.0.4") + install_python_packages 'pkgs[@]' + virtualenv /tmp/venv.UIQ6VEp17V New python executable in /tmp/venv.UIQ6VEp17V/bin/python2 Also creating executable in /tmp/venv.UIQ6VEp17V/bin/python Installing setuptools, pkg_resources, pip, wheel...done.
- [2] in jewel, we create venv in "make all" (see [3]) for preparing "make check", but
virtualenv
does not have access to network at that moment, probably that's why we are in this dilemma.
and pbuilder have no access to its host environment, that's why "BUILD_HOME" does not work.
The directory '/home/jenkins-build/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
we can either
- fulfill the requirement of virtualenv using
BINDMOUNTS
, tomount --bind
a host directory in the chroot env, a good candidates is "/home/jenkins-build/.cache/", but we might need to pre-populate this directory with required wheels for better performance. another candidate is a ccache directory shared among builds! - or, pre-populate the wheels in the wheelhouse directory as expected by setup-virtualenv.sh script. and put them into the dist tarball. so the build task can use them without the need to access network.
- or, enable the network access from pbuilder, to put "USENETWORK=yes" in
~/.pbuilderrc
. - or, add yet another workaround in setup-virtualenv.sh to tell if it is in a pbuild env, if it is, then do nothing and return. how? maybe we can check if
$HOME
is readable?
personally, i'd prefer option 2.
---
[1] success xenial build of ceph-11.0.0: https://jenkins.ceph.com/job/ceph-dev-build/655/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[2] failed xenial build of ceph-10.2.2: https://jenkins.ceph.com/job/ceph-dev-build/649/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[3]
$ git grep -w ceph-disk-all src/ceph-disk/Makefile.am:ceph-disk-all: ${CEPH_DISK_VIRTUALENV} src/ceph-disk/Makefile.am:LOCAL_ALL += ceph-disk-all
Updated by Alfredo Deza over 7 years ago
The " parent directory is not owned by the current user and the cache has been disabled" error is misleading. Setting the BUILD_HOME variable to `mktemp -d` gets rid of the error (see [0])
In this case, $HOME becomes readable and the error goes away. But the traceback persists because it is using the cached wheels directory provided by Xenial which has the
buggy version of pip that has that error (8.1.1)
Updated by Kefu Chai over 7 years ago
- Assignee changed from Kefu Chai to Alfredo Deza
Updated by Alfredo Deza over 7 years ago
- Status changed from New to Resolved
We are now having to allow network access when building. This sounds very unreasonable and it isn't something that happened before.
Pull Request to work around this issue in build scripts: https://github.com/ceph/ceph-build/pull/486
This issue would help prevent this in the future: #15260