Bug #16169: ceph-detect-init target fails for Jewel build - devops - Ceph

cd ./ceph-detect-init ; ../tools/setup-virtualenv.sh /tmp/ceph-detect-init-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-detect-init-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e .
New python executable in /tmp/ceph-detect-init-virtualenv/bin/python2.7
Also creating executable in /tmp/ceph-detect-init-virtualenv/bin/python
Installing setuptools, pkg_resources, pip, wheel...cd ./ceph-disk ; ../tools/setup-virtualenv.sh /tmp/ceph-disk-virtualenv ; test -d wheelhouse && export NO_INDEX=--no-index ; /tmp/ceph-disk-virtualenv/bin/pip install $NO_INDEX --use-wheel --find-links=file://$(pwd)/wheelhouse -e .
New python executable in /tmp/ceph-disk-virtualenv/bin/python2.7
Also creating executable in /tmp/ceph-disk-virtualenv/bin/python
Installing setuptools, pkg_resources, pip, wheel...  
Traceback (most recent call last):
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/commands/install.py", line 317, in run
    requirement_set.prepare_files(finder)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_set.py", line 512, in _prepare_file
    finder, self.upgrade, require_hashes)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/req/req_install.py", line 273, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 440, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 398, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 543, in _get_pages
    page = self._get_page(location)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 646, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/index.py", line 755, in get_page
    "Cache-Control": "max-age=600",
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/download.py", line 378, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/tmp/ceph-detect-init-virtualenv/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
----------------------------------------

I think that part of the error is that it is using Python's PIP from /usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl

Like I mentioned, this only happens in Jewel. Could you please take some time to investigate this? Maybe something needs to get backported/changed for the ceph-disk virtualenv setup for Jewel?

Actions

Copy link

#5

Updated by Loïc Dachary over 7 years ago

Assignee changed from Loïc Dachary to Kefu Chai

@Kefu Chai would you mind taking a look ? There has been two dozen changes since I last modified this part of the code about six months ago. Given the urgency of the fix, you are probably better equipped than I am. Thanks :-)

Actions

Copy link

#6

Updated by Dan Mick over 7 years ago

It seems like trying to mix /usr/share/python-wheels and a local wheelhouse is doomed to failure. The /usr/share version of requests does not include a packages directory with local packages, so gets the system urllib3 and chardet. I don't understand why the local wheelhouse is there, because the code doesn't have comments explaining it.

Actions

Copy link

#7

Updated by Nathan Cutler over 7 years ago

I seem to recall that the local wheelhouse is there to accommodate building in environments (like OBS) that aren't connected to the Internet.

Actions

Copy link

#8

Updated by Dan Mick over 7 years ago

I can't figure out what in the jenkins build is doing the same job as install-deps.sh does, to prepopulate wheelhouse. I don't yet understand how that should affect the installation of pip in the virtualenv, but certainly the code that tries to install wheels from wheelhouse is not working.

Actions

Copy link

#9

Updated by Loïc Dachary over 7 years ago

I think this may be another case of jenkins not cleaning after itself and starting a job that is under the influence of leftovers from a previous job. Or that jenkins is indeed installing pip from package and not providing the job with a pristine environment. There are two ways out of this: fixing jenkins or backporting one of the workarounds implemented in Ceph to compensate for jenkins shortcomings. I'm unfortunately not familiar enough with those workarounds to backport them quickly :-(

Actions

Copy link

#10

Updated by Alfredo Deza over 7 years ago

@Loïc Dachary I don't think this is a case of environmental leftovers from other builds. We can consistently replicate this problem for the Jewel branch of Ceph and in Xenial only.

Every other branch in Ceph for all the other combination of supported distribution passes. This is specific to just Jewel and Xenial.

Actions

Copy link

#11

Updated by Kefu Chai over 7 years ago

so we are using pbuilder to build the package, and pbuilder is basically a clean room created by chroot:

[1] in kraken, we don't create venv in "make all", so the only venv is created by ansible, so it has access to network.

Ubuntu is unknown, dependencies will have to be installed manually.
+ pkgs=("chacractl>=0.0.4")
+ install_python_packages 'pkgs[@]'
+ virtualenv /tmp/venv.UIQ6VEp17V
New python executable in /tmp/venv.UIQ6VEp17V/bin/python2
Also creating executable in /tmp/venv.UIQ6VEp17V/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

[2] in jewel, we create venv in "make all" (see [3]) for preparing "make check", but virtualenv does not have access to network at that moment, probably that's why we are in this dilemma.

and pbuilder have no access to its host environment, that's why "BUILD_HOME" does not work.

The directory '/home/jenkins-build/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

we can either

fulfill the requirement of virtualenv using BINDMOUNTS, to mount --bind a host directory in the chroot env, a good candidates is "/home/jenkins-build/.cache/", but we might need to pre-populate this directory with required wheels for better performance. another candidate is a ccache directory shared among builds!
or, pre-populate the wheels in the wheelhouse directory as expected by setup-virtualenv.sh script. and put them into the dist tarball. so the build task can use them without the need to access network.
or, enable the network access from pbuilder, to put "USENETWORK=yes" in ~/.pbuilderrc.
or, add yet another workaround in setup-virtualenv.sh to tell if it is in a pbuild env, if it is, then do nothing and return. how? maybe we can check if $HOME is readable?

personally, i'd prefer option 2.

---
[1] success xenial build of ceph-11.0.0: https://jenkins.ceph.com/job/ceph-dev-build/655/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[2] failed xenial build of ceph-10.2.2: https://jenkins.ceph.com/job/ceph-dev-build/649/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/consoleFull
[3]

$ git grep -w ceph-disk-all
src/ceph-disk/Makefile.am:ceph-disk-all: ${CEPH_DISK_VIRTUALENV}
src/ceph-disk/Makefile.am:LOCAL_ALL += ceph-disk-all

Actions

Copy link

#12

Updated by Alfredo Deza over 7 years ago

The " parent directory is not owned by the current user and the cache has been disabled" error is misleading. Setting the BUILD_HOME variable to `mktemp -d` gets rid of the error (see [0])

In this case, $HOME becomes readable and the error goes away. But the traceback persists because it is using the cached wheels directory provided by Xenial which has the
buggy version of pip that has that error (8.1.1)

[0] https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/682/console

Actions

Copy link

#13

Updated by Kefu Chai over 7 years ago

Assignee changed from Kefu Chai to Alfredo Deza

Actions

Copy link

#14

Updated by Alfredo Deza over 7 years ago

Status changed from New to Resolved

We are now having to allow network access when building. This sounds very unreasonable and it isn't something that happened before.

Pull Request to work around this issue in build scripts: https://github.com/ceph/ceph-build/pull/486

This issue would help prevent this in the future: #15260

Project

General

Profile

Ceph » devops

Custom queries

Bug #16169

ceph-detect-init target fails for Jewel build

Updated by Alfredo Deza almost 8 years ago

Updated by Loïc Dachary almost 8 years ago

Updated by Loïc Dachary almost 8 years ago

Updated by Alfredo Deza over 7 years ago

Updated by Loïc Dachary over 7 years ago

Updated by Dan Mick over 7 years ago

Updated by Nathan Cutler over 7 years ago

Updated by Dan Mick over 7 years ago

Updated by Loïc Dachary over 7 years ago

Updated by Alfredo Deza over 7 years ago

Updated by Kefu Chai over 7 years ago

Updated by Alfredo Deza over 7 years ago

Updated by Kefu Chai over 7 years ago

Updated by Alfredo Deza over 7 years ago