Bug #58643
closedragweed fails with ModuleNotFoundError (No module named 'rados')
0%
Description
2023-02-05T15:08:30.417 DEBUG:teuthology.orchestra.run.smithi170:ragweed tests against rgw> RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw' 2023-02-05T15:08:30.685 INFO:teuthology.orchestra.run.smithi170.stderr:Failure: ModuleNotFoundError (No module named 'rados') ... ERROR 2023-02-05T15:08:30.685 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-02-05T15:08:30.685 INFO:teuthology.orchestra.run.smithi170.stderr:====================================================================== 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr:ERROR: Failure: ModuleNotFoundError (No module named 'rados') 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr:---------------------------------------------------------------------- 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr:Traceback (most recent call last): 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/virtualenv/lib64/python3.6/site-packages/nose/failure.py", line 39, in runTest 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr: raise self.exc_val.with_traceback(self.tb) 2023-02-05T15:08:30.686 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/virtualenv/lib64/python3.6/site-packages/nose/loader.py", line 418, in loadTestsFromName 2023-02-05T15:08:30.687 INFO:teuthology.orchestra.run.smithi170.stderr: addr.filename, addr.module) 2023-02-05T15:08:30.687 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/virtualenv/lib64/python3.6/site-packages/nose/importer.py", line 47, in importFromPath 2023-02-05T15:08:30.687 INFO:teuthology.orchestra.run.smithi170.stderr: return self.importFromDir(dir_path, fqname) 2023-02-05T15:08:30.687 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/virtualenv/lib64/python3.6/site-packages/nose/importer.py", line 94, in importFromDir 2023-02-05T15:08:30.687 INFO:teuthology.orchestra.run.smithi170.stderr: mod = load_module(part_fqname, fh, filename, desc) 2023-02-05T15:08:30.688 INFO:teuthology.orchestra.run.smithi170.stderr: File "/usr/lib64/python3.6/imp.py", line 235, in load_module 2023-02-05T15:08:30.688 INFO:teuthology.orchestra.run.smithi170.stderr: return load_source(name, filename, file) 2023-02-05T15:08:30.688 INFO:teuthology.orchestra.run.smithi170.stderr: File "/usr/lib64/python3.6/imp.py", line 172, in load_source 2023-02-05T15:08:30.688 INFO:teuthology.orchestra.run.smithi170.stderr: module = _load(spec) 2023-02-05T15:08:30.688 INFO:teuthology.orchestra.run.smithi170.stderr: File "<frozen importlib._bootstrap>", line 684, in _load 2023-02-05T15:08:30.689 INFO:teuthology.orchestra.run.smithi170.stderr: File "<frozen importlib._bootstrap>", line 665, in _load_unlocked 2023-02-05T15:08:30.689 INFO:teuthology.orchestra.run.smithi170.stderr: File "<frozen importlib._bootstrap_external>", line 678, in exec_module 2023-02-05T15:08:30.689 INFO:teuthology.orchestra.run.smithi170.stderr: File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed 2023-02-05T15:08:30.689 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/tests/tests.py", line 2, in <module> 2023-02-05T15:08:30.689 INFO:teuthology.orchestra.run.smithi170.stderr: import ragweed.framework 2023-02-05T15:08:30.690 INFO:teuthology.orchestra.run.smithi170.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 14, in <module> 2023-02-05T15:08:30.690 INFO:teuthology.orchestra.run.smithi170.stderr: import rados 2023-02-05T15:08:30.690 INFO:teuthology.orchestra.run.smithi170.stderr:ModuleNotFoundError: No module named 'rados' 2023-02-05T15:08:30.690 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-02-05T15:08:30.690 INFO:teuthology.orchestra.run.smithi170.stderr:---------------------------------------------------------------------- 2023-02-05T15:08:30.691 INFO:teuthology.orchestra.run.smithi170.stderr:Ran 1 test in 0.053s 2023-02-05T15:08:30.691 INFO:teuthology.orchestra.run.smithi170.stderr: 2023-02-05T15:08:30.691 INFO:teuthology.orchestra.run.smithi170.stderr:FAILED (errors=1)
examples:
http://qa-proxy.ceph.com/teuthology/gsalomon-2023-02-05_14:42:52-rgw:verify-json_rgw_integration-distro-default-smithi/7149177/teuthology.log
http://qa-proxy.ceph.com/teuthology/cbodley-2023-02-03_15:44:47-rgw-wip-cbodley-testing-distro-default-smithi/7146943/teuthology.log
Updated by Casey Bodley about 1 year ago
Updated by Laura Flores about 1 year ago
@Casey I have not seen this in the rados suite yet, but I will keep an eye out.
Updated by Laura Flores about 1 year ago
- Translation missing: en.field_tag_list set to test-failure
Updated by Casey Bodley about 1 year ago
i'm seeing the failures under rgw/verify (which only runs against centos8) and rgw/multifs (which only runs against ubuntu), so it isn't specific to either distro
Updated by Mark Kogan about 1 year ago
reviewing the mentioned teuthology.log files,
saw something out of the ordinary in both:
for demonstration lets take: http://qa-proxy.ceph.com/teuthology/cbodley-2023-02-03_15:44:47-rgw-wip-cbodley-testing-distro-default-smithi/7146943/teuthology.log
searching this file for `python3-r`
shows that
2023-02-04T15:10:46.600 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:18.0.0-2056.g12d02ca9.el8 ceph 371 k 2023-02-04T15:10:46.600 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:18.0.0-2056.g12d02ca9.el8 ceph 363 k 2023-02-04T15:10:46.600 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:18.0.0-2056.g12d02ca9.el8 ceph 107 k
are downloaded
BUT later only
2023-02-04T15:10:52.716 INFO:teuthology.orchestra.run.smithi159.stdout:(21/105): python3-rbd-18.0.0-2056.g12d02ca9.el8 6.1 MB/s | 363 kB 00:00 2023-02-04T15:10:52.767 INFO:teuthology.orchestra.run.smithi159.stdout:(23/105): python3-rgw-18.0.0-2056.g12d02ca9.el8 2.1 MB/s | 107 kB 00:00
are installed !
if `python3-rados` rpm is NOT getting installed, it would explain the error:
ERROR: Failure: ModuleNotFoundError (No module named 'rados')
Updated by Casey Bodley about 1 year ago
from that log, i see 'python3-rados' during the download step
(20/105): python3-rados-18.0.0-2056.g12d02ca9.e
then during the install step, it's called 'python3-rados-2'
Installing : python3-rados-2:18.0.0-2056.g12d02ca9.el8.x86_64 8/107
i compared this against a successful run from january, and saw the same pattern:
Updated by Casey Bodley about 1 year ago
looking back at ragweed PRs, this one now looks suspicious because it removed the --system-site-packages flag:
https://github.com/ceph/ragweed/pull/23
that worked when i tested it, so i assumed the failure wasn unrelated. since that PR was only necessary for ubuntu22, i'll revert it from the ceph-* branches. we're replacing virtualenv+nose with tox+pytest there anyway
Updated by Casey Bodley about 1 year ago
tried scheduling a job against the updated ceph-master branch in https://pulpito.ceph.com/cbodley-2023-02-15_15:03:45-rgw-wip-cbodley-testing-distro-default-smithi/
Updated by Casey Bodley about 1 year ago
- Status changed from New to Resolved
- Tags set to ragweed
ragweed passed in http://qa-proxy.ceph.com/teuthology/cbodley-2023-02-15_15:03:45-rgw-wip-cbodley-testing-distro-default-smithi/7174864/teuthology.log
thanks to everyone that helped with this!
Updated by Laura Flores about 1 year ago
@Casey how exactly was this fixed? Something was reverted?
Updated by Casey Bodley about 1 year ago
Laura Flores wrote:
@Casey how exactly was this fixed? Something was reverted?
correct, i removed the commit from https://github.com/ceph/ragweed/pull/23 all of its release branches