Bug #43192
closedmgr/dashboard: Teuthology failure: 'module' object has no attribute 'test_auth'
0%
Description
The suite tasks.mgr.dashboard.test_auth failed in the rados QA run.Sage mentioned that https://github.com/ceph/ceph/pull/31937 might be related (latest QA run of this PR passed the rados/dashboard suite: http://pulpito.ceph.com/laura-2019-12-05_07:03:25-rados:dashboard-wip-laura-testing-31937-distro-basic-mira/)
Test failures:
- http://pulpito.ceph.com/sage-2019-12-08_09:42:22-rados-wip-sage-testing-2019-12-07-2341-distro-basic-smithi/
- http://pulpito.ceph.com/tdehler-2019-12-10_06:31:23-rados:dashboard-master-distro-basic-mira/
- http://pulpito.ceph.com/laura-2019-12-10_07:58:09-rados:dashboard-wip-laura-testing-31851-2-distro-basic-mira/
This is currently causing all QA runs (master + PR on top) to fail
Also happening in Nautilus: http://pulpito.ceph.com/sage-2019-12-08_05:43:33-rados-nautilus-distro-basic-smithi/
(PR https://github.com/ceph/ceph/pull/31937 was not backported, so it's not related)
2019-12-08T10:27:51.403 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-07-2341/qa/tasks/cephfs_test_runner.py", line 162, in task
module_suites.append(decorating_loader.loadTestsFromName(mod_name))
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-07-2341/qa/tasks/cephfs_test_runner.py", line 32, in loadTestsFromName
result = super(DecoratingLoader, self).loadTestsFromName(name, module)
File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_auth'
2019-12-08T10:27:51.408 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2019-12-08T10:27:51.421 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2019-12-08T10:27:51.437 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-07-2341/qa/tasks/rgw.py", line 396, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-07-2341/qa/tasks/cephfs_test_runner.py", line 162, in task
module_suites.append(decorating_loader.loadTestsFromName(mod_name))
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2019-12-07-2341/qa/tasks/cephfs_test_runner.py", line 32, in loadTestsFromName
result = super(DecoratingLoader, self).loadTestsFromName(name, module)
File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_auth'
2019-12-08T10:27:51.439 DEBUG:tasks.rgw.client.0:waiting for process to exit
Updated by Volker Theile over 4 years ago
Can not reproduce it by running
./run-backend-api-tests.sh tasks.mgr.dashboard.test_auth
in a vstart environment.
2019-12-09 09:22:12,199.199 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_auth.AuthTest.test_unauthorized'] 2019-12-09 09:22:13,195.195 INFO:__main__:test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) ... ok 2019-12-09 09:22:13,195.195 INFO:__main__:Stopped test: test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) in 1.029999s 2019-12-09 09:22:13,196.196 INFO:__main__: 2019-12-09 09:22:13,196.196 INFO:__main__:---------------------------------------------------------------------- 2019-12-09 09:22:13,196.196 INFO:__main__:Ran 11 tests in 84.897s 2019-12-09 09:22:13,196.196 INFO:__main__: 2019-12-09 09:22:13,196.196 INFO:__main__:OK Name Stmts Miss Cover ------------------------------------------------------------------------------- /ceph/src/pybind/mgr/dashboard/__init__.py 22 20 9% /ceph/src/pybind/mgr/dashboard/controllers/__init__.py 553 541 2% /ceph/src/pybind/mgr/dashboard/controllers/auth.py 38 16 58% /ceph/src/pybind/mgr/dashboard/controllers/host.py 61 52 15% /ceph/src/pybind/mgr/dashboard/exceptions.py 57 45 21% /ceph/src/pybind/mgr/dashboard/module.py 260 259 1% /ceph/src/pybind/mgr/dashboard/plugins/__init__.py 40 38 5% /ceph/src/pybind/mgr/dashboard/plugins/debug.py 32 28 13% /ceph/src/pybind/mgr/dashboard/plugins/feature_toggles.py 70 63 10% /ceph/src/pybind/mgr/dashboard/plugins/pluggy.py 37 35 5% /ceph/src/pybind/mgr/dashboard/plugins/plugin.py 15 14 7% /ceph/src/pybind/mgr/dashboard/services/access_control.py 513 480 6% /ceph/src/pybind/mgr/dashboard/services/auth.py 131 51 61% /ceph/src/pybind/mgr/dashboard/services/exception.py 88 66 25% /ceph/src/pybind/mgr/dashboard/services/orchestrator.py 75 58 23% /ceph/src/pybind/mgr/dashboard/settings.py 102 98 4% /ceph/src/pybind/mgr/dashboard/tools.py 531 450 15% /ceph/src/pybind/mgr/mgr_util.py 92 88 4% /ceph/src/pybind/mgr/orchestrator.py 601 589 2% ------------------------------------------------------------------------------- TOTAL 3318 2991 10%
Updated by Tatjana Dehler over 4 years ago
I'm seeing the failure here as well: http://pulpito.ceph.com/tdehler-2019-12-10_06:31:23-rados:dashboard-master-distro-basic-mira/
And yes, I can confirm that the issue doesn't happen on vstart.
Updated by Laura Paduano over 4 years ago
- Priority changed from Normal to High
- Affected Versions v14.2.5 added
Updated by Laura Paduano over 4 years ago
I also can not reproduce it in vstart when running
./run-backend-api-tests.sh tasks.mgr.dashboard.test_auth
in my local environment
Updated by Laura Paduano over 4 years ago
Tatjana mentioned if https://github.com/ceph/teuthology/pull/1367 could be related (?)
Updated by Kefu Chai over 4 years ago
qa/tasks/cephfs_test_runner.py
tries to load the tests listed by "cephfs_test_runner/modules" using __import__
. when it tries to load tasks.mgr.test_dashboard
, everything goes fine. but when it comes to tasks.mgr.dashboard.test_auth
, we have an ImportError
of "No module named jwt".
because test_auth.py has
import jwt
but this line of code was introduced back in 2018 in 2f5e7c33925c51c56b074453c50c4286375333eb. and we have
%if 0%{?fedora} || 0%{?rhel} Requires: python%{_python_buildid}-cherrypy Requires: python%{_python_buildid}-ipaddress Requires: python%{_python_buildid}-jwt Requires: python%{_python_buildid}-routes Requires: python%{_python_buildid}-werkzeug %endif
in ceph.spec.in. so ceph-mgr-dashboard should depend on python-jwt.
but the funny thing is, this package was not installed:
$ grep ceph-mgr-dashboard /a//kchai-2019-12-11_19:03:22-rados:dashboard-master-distro-basic-mira/4593279/teuthology.log|grep Dependency 2019-12-11T19:25:27.273 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch 2019-12-11T19:25:27.382 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch 2019-12-11T19:25:28.734 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch 2019-12-11T19:25:28.843 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch 2019-12-11T19:30:06.543 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch 2019-12-11T19:30:06.654 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch $ grep python-jwt /a//kchai-2019-12-11_19:03:22-rados:dashboard-master-distro-basic-mira/4593279/teuthology.log || echo bummer bummer
i checked the dependency relationship of ceph-mgr-dashboard rpm package:
$ curl --silent --remote-name --location 'https://1.chacra.ceph.com/r/ceph/master/be58eb0a8d242bb4542177a1278be8ea1ba04045/centos/7/flavors/default/noarch/ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch.rpm' $ rpm -qpR ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch.rpm /bin/sh /bin/sh ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7 ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7 pyOpenSSL python-cherrypy python-ipaddress python-jwt python-routes python-werkzeug rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PartialHardlinkSets) <= 4.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1
it does depends on python-jwt. the crux of the problem is, as implied by the following backtrace
2019-12-11T19:29:39.399 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 214, in task module_suites.append(decorating_loader.loadTestsFromName(mod_name)) File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 84, in loadTestsFromName result = self._loadTestsFromName(name, module) File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 57, in _loadTestsFromName parent, obj = obj, getattr(obj, part) AttributeError: 'module' object has no attribute 'test_auth'
teuthology-worker runs on "teuthology" node, where it clones the qa suite repo. and this machine does not have python-jwt installed
kchai@teuthology:~$ python Python 2.7.12 (default, Oct 8 2019, 14:14:10) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import jwt Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named jwt $ dpkg -s python-jwt dpkg-query: package 'python-jwt' is not installed and no information is available Use dpkg --info (= dpkg-deb --info) to examine archive files, and dpkg --contents (= dpkg-deb --contents) to list their contents.
i am not able to query the log of apt
$ zgrep jwt /var/log/apt/* gzip: /var/log/apt/term.log: Permission denied gzip: /var/log/apt/term.log.10.gz: Permission denied gzip: /var/log/apt/term.log.11.gz: Permission denied gzip: /var/log/apt/term.log.12.gz: Permission denied gzip: /var/log/apt/term.log.1.gz: Permission denied gzip: /var/log/apt/term.log.2.gz: Permission denied gzip: /var/log/apt/term.log.3.gz: Permission denied gzip: /var/log/apt/term.log.4.gz: Permission denied gzip: /var/log/apt/term.log.5.gz: Permission denied gzip: /var/log/apt/term.log.6.gz: Permission denied gzip: /var/log/apt/term.log.7.gz: Permission denied gzip: /var/log/apt/term.log.8.gz: Permission denied gzip: /var/log/apt/term.log.9.gz: Permission denied
Updated by Kefu Chai over 4 years ago
i was wrong, teuthworker is using its own venv
ps aux |grep teuthworker| head -n1 teuthwo+ 1385 1.0 0.2 191844 79616 ? S 03:36 0:12 /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/python /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology -v --lock --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-12-11_23:13:54-multimds-wip-pdonnell-testing-20191211.205606-distro-basic-smithi/4593804 --name pdonnell-2019-12-11_23:13:54-multimds-wip-pdonnell-testing-20191211.205606-distro-basic-smithi --description multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_7.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} -- /tmp/teuthology-worker.wFUR2t.tmp kchai@teuthology:~/teuthology$ source /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/activate (virtualenv) kchai@teuthology:~/teuthology$ python Python 2.7.12 (default, Oct 8 2019, 14:14:10) [GCC 5.4.0 20160609] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import jwt Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named jwt >>>
and teuthology/requirements.txt does not contain "PyJWT". so this is expected.
seems the right solution is to add "PyJWT" as an extra_require in setup.py,
Updated by Kefu Chai over 4 years ago
- Project changed from mgr to teuthology
- Category changed from 151 to Core
- Status changed from New to Fix Under Review
- Assignee set to Kefu Chai
Updated by Kefu Chai over 4 years ago
- Status changed from Fix Under Review to Resolved