Actions
Bug #49017
closedragweed bootstrap failing in teuthology
Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
ragweed
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
example from current master branch:
http://qa-proxy.ceph.com/teuthology/teuthology-2021-01-23_03:05:04-rgw-master-distro-basic-gibba/5818286/teuthology.log
2021-01-23T04:36:50.916 INFO:teuthology.orchestra.run.gibba028.stdout:Installing collected packages: pycparser, cffi
2021-01-23T04:36:50.916 INFO:teuthology.orchestra.run.gibba028.stdout: Running setup.py install for pycparser: started
2021-01-23T04:36:51.121 INFO:teuthology.orchestra.run.gibba028.stdout: Running setup.py install for pycparser: finished with status 'done'
2021-01-23T04:36:51.123 INFO:teuthology.orchestra.run.gibba028.stdout: Running setup.py install for cffi: started
2021-01-23T04:36:52.454 INFO:teuthology.orchestra.run.gibba028.stdout: Running setup.py install for cffi: finished with status 'done'
2021-01-23T04:36:52.467 INFO:teuthology.orchestra.run.gibba028.stdout:Successfully installed cffi-0.8.6 pycparser-2.10
2021-01-23T04:36:53.201 INFO:teuthology.orchestra.run.gibba028.stdout:Already using interpreter /usr/bin/python3
2021-01-23T04:36:53.202 INFO:teuthology.orchestra.run.gibba028.stdout:Using base prefix '/usr'
2021-01-23T04:36:53.202 INFO:teuthology.orchestra.run.gibba028.stdout:New python executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python3
2021-01-23T04:36:53.202 INFO:teuthology.orchestra.run.gibba028.stdout:Also creating executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python
2021-01-23T04:36:53.490 INFO:teuthology.orchestra.run.gibba028.stdout:Installing setuptools, pkg_resources, pip, wheel...
2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout: Complete output from command /home/ubuntu/cephtes...rtualenv/bin/python3 - setuptools pkg_resources pip wheel:
2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout: Traceback (most recent call last):
2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 33, in vendored
2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout:ModuleNotFoundError: No module named 'pip._vendor.cachecontrol'
2021-01-23T04:36:53.492 INFO:teuthology.orchestra.run.gibba028.stdout:
2021-01-23T04:36:53.492 INFO:teuthology.orchestra.run.gibba028.stdout:During handling of the above exception, another exception occurred:
2021-01-23T04:36:53.492 INFO:teuthology.orchestra.run.gibba028.stdout:
2021-01-23T04:36:53.492 INFO:teuthology.orchestra.run.gibba028.stdout:Traceback (most recent call last):
2021-01-23T04:36:53.492 INFO:teuthology.orchestra.run.gibba028.stdout: File "<stdin>", line 7, in <module>
2021-01-23T04:36:53.493 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.493 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.493 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.493 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.493 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/__init__.py", line 22, in <module>
2021-01-23T04:36:53.494 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.494 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.494 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.494 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.495 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 64, in <module>
2021-01-23T04:36:53.495 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 36, in vendored
2021-01-23T04:36:53.495 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.495 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.496 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.496 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.496 INFO:teuthology.orchestra.run.gibba028.stdout: File "/home/ubuntu/cephtest/ragweed/virtualenv/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
2021-01-23T04:36:53.496 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.496 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.497 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.497 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.497 INFO:teuthology.orchestra.run.gibba028.stdout: File "/home/ubuntu/cephtest/ragweed/virtualenv/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
2021-01-23T04:36:53.497 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.497 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.498 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.498 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.498 INFO:teuthology.orchestra.run.gibba028.stdout: File "/home/ubuntu/cephtest/ragweed/virtualenv/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 4, in <module>
2021-01-23T04:36:53.498 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.498 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.499 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.499 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.499 INFO:teuthology.orchestra.run.gibba028.stdout: File "/home/ubuntu/cephtest/ragweed/virtualenv/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/__init__.py", line 84, in <module>
2021-01-23T04:36:53.499 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-01-23T04:36:53.500 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-01-23T04:36:53.500 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
2021-01-23T04:36:53.500 INFO:teuthology.orchestra.run.gibba028.stdout: File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
2021-01-23T04:36:53.500 INFO:teuthology.orchestra.run.gibba028.stdout: File "/home/ubuntu/cephtest/ragweed/virtualenv/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 46, in <module>
2021-01-23T04:36:53.500 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
2021-01-23T04:36:53.501 INFO:teuthology.orchestra.run.gibba028.stdout: from OpenSSL import crypto, SSL
2021-01-23T04:36:53.501 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 12, in <module>
2021-01-23T04:36:53.501 INFO:teuthology.orchestra.run.gibba028.stdout: from cryptography import x509
2021-01-23T04:36:53.501 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/cryptography/x509/__init__.py", line 8, in <module>
2021-01-23T04:36:53.502 INFO:teuthology.orchestra.run.gibba028.stdout: from cryptography.x509.base import (
2021-01-23T04:36:53.502 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/cryptography/x509/base.py", line 16, in <module>
2021-01-23T04:36:53.502 INFO:teuthology.orchestra.run.gibba028.stdout: from cryptography.x509.extensions import Extension, ExtensionType
2021-01-23T04:36:53.502 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/cryptography/x509/extensions.py", line 18, in <module>
2021-01-23T04:36:53.502 INFO:teuthology.orchestra.run.gibba028.stdout: from cryptography.hazmat.primitives import constant_time, serialization
2021-01-23T04:36:53.503 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/constant_time.py", line 9, in <module>
2021-01-23T04:36:53.503 INFO:teuthology.orchestra.run.gibba028.stdout: from cryptography.hazmat.bindings._constant_time import lib
2021-01-23T04:36:53.503 INFO:teuthology.orchestra.run.gibba028.stdout:AttributeError: module '_cffi_backend' has no attribute '_init_cffi_1_0_external_module'
2021-01-23T04:36:53.503 INFO:teuthology.orchestra.run.gibba028.stdout:----------------------------------------
2021-01-23T04:36:53.503 INFO:teuthology.orchestra.run.gibba028.stdout:...Installing setuptools, pkg_resources, pip, wheel...done.
2021-01-23T04:36:53.504 INFO:teuthology.orchestra.run.gibba028.stderr:Traceback (most recent call last):
2021-01-23T04:36:53.504 INFO:teuthology.orchestra.run.gibba028.stderr: File "/usr/bin/virtualenv", line 11, in <module>
2021-01-23T04:36:53.504 INFO:teuthology.orchestra.run.gibba028.stderr: load_entry_point('virtualenv==15.1.0', 'console_scripts', 'virtualenv')()
2021-01-23T04:36:53.505 INFO:teuthology.orchestra.run.gibba028.stderr: File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
2021-01-23T04:36:53.505 INFO:teuthology.orchestra.run.gibba028.stderr: symlink=options.symlink)
2021-01-23T04:36:53.505 INFO:teuthology.orchestra.run.gibba028.stderr: File "/usr/lib/python3/dist-packages/virtualenv.py", line 992, in create_environment
2021-01-23T04:36:53.505 INFO:teuthology.orchestra.run.gibba028.stderr: download=download,
2021-01-23T04:36:53.506 INFO:teuthology.orchestra.run.gibba028.stderr: File "/usr/lib/python3/dist-packages/virtualenv.py", line 922, in install_wheel
2021-01-23T04:36:53.506 INFO:teuthology.orchestra.run.gibba028.stderr: call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
2021-01-23T04:36:53.506 INFO:teuthology.orchestra.run.gibba028.stderr: File "/usr/lib/python3/dist-packages/virtualenv.py", line 817, in call_subprocess
2021-01-23T04:36:53.506 INFO:teuthology.orchestra.run.gibba028.stderr: % (cmd_desc, proc.returncode))
2021-01-23T04:36:53.506 INFO:teuthology.orchestra.run.gibba028.stderr:OSError: Command /home/ubuntu/cephtes...rtualenv/bin/python3 - setuptools pkg_resources pip wheel failed with error code 1
Updated by Casey Bodley about 3 years ago
https://pulpito.ceph.com/teuthology-2021-01-23_03:05:04-rgw-master-distro-basic-gibba/
it looks like the failures are only on ubuntu. the rgw/verify suite all passed against centos
Updated by Casey Bodley about 3 years ago
from logs, i see that
Ubuntu 18.04.5 is installing libffi-dev:amd64 (3.2.1-8), and
Centos 8.2 is installing libffi-devel-3.1-22.el8
but i can't tell whether the issue is this libffi version, or in some python gorp in between
Updated by Mark Kogan about 3 years ago
seems that the pip version is from python 2.7 instead of 3
File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 33, in vendored
from ubuntu pod:
podman run -it --rm --net=host ubuntu:18.04 apt update apt install python-pip ls -l /usr/share/python-wheels/pip* -rw-r--r-- 1 root root 159285 Oct 22 14:40 /usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl pip --version pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
on fedora
ls -l /usr/share/python-wheels/pip* .rwxr-xr-x root root 1.3 MB Tue Aug 11 18:37:28 2020 /usr/share/python-wheels/pip-20.2.2-py2.py3-none-any.whl
checking how to switch to python3 on ubuntu
Updated by Casey Bodley about 3 years ago
as a workaround, the 'parallel-no-cephadm' upgrade suite was pinned to centos in https://github.com/ceph/ceph/pull/39066
we'll need to remember to unpin it once this is resolved
Updated by Mark Kogan about 3 years ago
- Status changed from New to Fix Under Review
2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout: File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 33, in vendored 2021-01-23T04:36:53.491 INFO:teuthology.orchestra.run.gibba028.stdout:ModuleNotFoundError: No module named 'pip._vendor.cachecontrol'
narrowing - the error is caused by missing
'cachecontrol' python module on some of the ubuntu machines (gibba),
No module named 'pip._vendor.cachecontrol'
adding it to the modules installed by bootstrap.
@ https://github.com/ceph/ragweed/pull/17
and monitoring gibba builds.
(fix is currently on ceph-master & ceph-octopus)
Updated by Yuri Weinstein about 3 years ago
maybe gibba specific @david
Updated by Casey Bodley about 3 years ago
- Status changed from Fix Under Review to Resolved
it looks like this is passing again on ubuntu according to https://pulpito.ceph.com/cbodley-2021-02-09_18:52:29-rgw:multifs-wip-cbodley-testing-distro-basic-gibba/
the fixes to ceph-octopus and ceph-pacific as well. thanks Mark! please update if there are any more failures
Updated by Mark Kogan almost 3 years ago
- Copied to Bug #50553: ragweed bootstrap failing in teuthology: "virtualenv: error: unrecognized arguments: --distribute" added
Actions