Infrastructure

According to the urllib3 changelog, there was a new version released on May 3rd, right around the time the above issue was reported:
https://urllib3.readthedocs.io/en/latest/changelog.html

Before May 3rd, we were installing urllib3 1.26.15, which supports appengine. Now, we are installing the latest version, urllib3 2.0.2. From urllib3 2.0.0 and on, appengine support was dropped, which is why we're seeing this Traceback.

There's more to the story though. urllib3 has two package dependencies, "requests" and "requests-toolbelt". In the builds linked above, we are installing the latest version of requests (2.30.0) which supports urllib3 2.0: https://pyup.io/packages/pypi/requests/changelog. So, that's good for us.

However, we are still installing an older version of requests-toolbelt (0.9.1), which doesn't yet support urllib3 2.0: https://github.com/requests/toolbelt/blob/master/HISTORY.rst. Only in the latest version of requests-toolbelt (1.0) does it have urllib3 2.0 support. This means that we are trying to import appengine since requests-toolbelt thinks urllib3 still supports it when it doesn't.

Solution: Install the latest version of requests-toolbelt, which is 1.0.0.

https://github.com/ceph/ceph-build/blob/7f3194a3a6f5681dc50e3a35c639a46f957b4f36/scripts/build_utils.sh#L232-L240

    pkgs=("${!1}")
    for package in ${pkgs[@]}; do
        echo $package
        # download packages to the local pip cache
        pip_download $venv $package
        # install packages from the local pip cache, ignoring pypi
        $venv/pip install --upgrade --exists-action=i --find-links="file://$PIP_SDIST_INDEX" --no-index $package
    done
}

Interesting. It looks like we only upgrade the packages that are set in the pkgs variable, which, from the failing builds, only contains one package: pkgs=("chacractl>=0.0.21")

    pkgs=("${!1}")
    for package in ${pkgs[@]}; do
        echo $package
        # download packages to the local pip cache
        pip_download $venv $package
        # install packages from the local pip cache, ignoring pypi
        $venv/pip install --upgrade --exists-action=i --find-links="file://$PIP_SDIST_INDEX" --no-index $package
    done
}

And when pip installs chacractl version 0.0.21, there is a constraint put on requests_toolbelt:

+ pip_download /tmp/venv.hMQnl0FKN1/bin 'chacractl>=0.0.21'
+ local venv=/tmp/venv.hMQnl0FKN1/bin
+ shift
+ local 'package=chacractl>=0.0.21'
+ shift
+ local options=
+ /tmp/venv.hMQnl0FKN1/bin/pip download --dest=/home/jenkins-build/.cache/pip 'chacractl>=0.0.21'
Collecting chacractl>=0.0.21
  File was already downloaded /home/jenkins-build/.cache/pip/chacractl-0.0.24-py3-none-any.whl
Collecting requests
  File was already downloaded /home/jenkins-build/.cache/pip/requests-2.30.0-py3-none-any.whl
Collecting tambo>=0.1.0
  File was already downloaded /home/jenkins-build/.cache/pip/tambo-0.4.0.tar.gz
Collecting requests-toolbelt==0.9.1
  File was already downloaded /home/jenkins-build/.cache/pip/requests_toolbelt-0.9.1-py2.py3-none-any.whl
Collecting idna<4,>=2.5
  File was already downloaded /home/jenkins-build/.cache/pip/idna-3.4-py3-none-any.whl
Collecting charset-normalizer<4,>=2
  File was already downloaded /home/jenkins-build/.cache/pip/charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting urllib3<3,>=1.21.1
  File was already downloaded /home/jenkins-build/.cache/pip/urllib3-2.0.2-py3-none-any.whl
Collecting certifi>=2017.4.17

I was able to reproduce this in an ubuntu:latest container. The issue is with chacractl. Even with the latest chacractl version of 0.0.24, it puts a constraint on requests-toolbelt to be the older version:

root@5bbf05b59ac4:~# python3 -m pip download chacractl        
Collecting chacractl
  Downloading chacractl-0.0.24-py3-none-any.whl (16 kB)
Collecting requests-toolbelt==0.9.1
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting tambo>=0.1.0
  Downloading tambo-0.4.0.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
Collecting requests
  File was already downloaded /root/requests-2.30.0-py3-none-any.whl
Collecting certifi>=2017.4.17
  File was already downloaded /root/certifi-2022.12.7-py3-none-any.whl
Collecting urllib3<3,>=1.21.1
  File was already downloaded /root/urllib3-2.0.2-py3-none-any.whl
Collecting idna<4,>=2.5
  File was already downloaded /root/idna-3.4-py3-none-any.whl
Collecting charset-normalizer<4,>=2
  File was already downloaded /root/charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./chacractl-0.0.24-py3-none-any.whl
Saved ./requests_toolbelt-0.9.1-py2.py3-none-any.whl
Saved ./tambo-0.4.0.tar.gz
Successfully downloaded chacractl requests-toolbelt requests tambo certifi charset-normalizer idna urllib3

Proposed a fix here: https://github.com/ceph/ceph-build/pull/2130