Tools » teuthology

Can not reproduce it by running

./run-backend-api-tests.sh tasks.mgr.dashboard.test_auth

2019-12-09 09:22:12,199.199 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_auth.AuthTest.test_unauthorized']
2019-12-09 09:22:13,195.195 INFO:__main__:test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) ... ok
2019-12-09 09:22:13,195.195 INFO:__main__:Stopped test: test_unauthorized (tasks.mgr.dashboard.test_auth.AuthTest) in 1.029999s
2019-12-09 09:22:13,196.196 INFO:__main__:
2019-12-09 09:22:13,196.196 INFO:__main__:----------------------------------------------------------------------
2019-12-09 09:22:13,196.196 INFO:__main__:Ran 11 tests in 84.897s
2019-12-09 09:22:13,196.196 INFO:__main__:
2019-12-09 09:22:13,196.196 INFO:__main__:OK
Name                                                        Stmts   Miss  Cover
-------------------------------------------------------------------------------
/ceph/src/pybind/mgr/dashboard/__init__.py                     22     20     9%
/ceph/src/pybind/mgr/dashboard/controllers/__init__.py        553    541     2%
/ceph/src/pybind/mgr/dashboard/controllers/auth.py             38     16    58%
/ceph/src/pybind/mgr/dashboard/controllers/host.py             61     52    15%
/ceph/src/pybind/mgr/dashboard/exceptions.py                   57     45    21%
/ceph/src/pybind/mgr/dashboard/module.py                      260    259     1%
/ceph/src/pybind/mgr/dashboard/plugins/__init__.py             40     38     5%
/ceph/src/pybind/mgr/dashboard/plugins/debug.py                32     28    13%
/ceph/src/pybind/mgr/dashboard/plugins/feature_toggles.py      70     63    10%
/ceph/src/pybind/mgr/dashboard/plugins/pluggy.py               37     35     5%
/ceph/src/pybind/mgr/dashboard/plugins/plugin.py               15     14     7%
/ceph/src/pybind/mgr/dashboard/services/access_control.py     513    480     6%
/ceph/src/pybind/mgr/dashboard/services/auth.py               131     51    61%
/ceph/src/pybind/mgr/dashboard/services/exception.py           88     66    25%
/ceph/src/pybind/mgr/dashboard/services/orchestrator.py        75     58    23%
/ceph/src/pybind/mgr/dashboard/settings.py                    102     98     4%
/ceph/src/pybind/mgr/dashboard/tools.py                       531    450    15%
/ceph/src/pybind/mgr/mgr_util.py                               92     88     4%
/ceph/src/pybind/mgr/orchestrator.py                          601    589     2%
-------------------------------------------------------------------------------
TOTAL                                                        3318   2991    10%

I'm seeing the failure here as well: http://pulpito.ceph.com/tdehler-2019-12-10_06:31:23-rados:dashboard-master-distro-basic-mira/

And yes, I can confirm that the issue doesn't happen on vstart.

I also can not reproduce it in vstart when running

./run-backend-api-tests.sh tasks.mgr.dashboard.test_auth

in my local environment

Tatjana mentioned if https://github.com/ceph/teuthology/pull/1367 could be related (?)

qa/tasks/cephfs_test_runner.py tries to load the tests listed by "cephfs_test_runner/modules" using __import__. when it tries to load tasks.mgr.test_dashboard, everything goes fine. but when it comes to tasks.mgr.dashboard.test_auth, we have an ImportError of "No module named jwt".

because test_auth.py has

import jwt

but this line of code was introduced back in 2018 in 2f5e7c33925c51c56b074453c50c4286375333eb. and we have

%if 0%{?fedora} || 0%{?rhel}
Requires:       python%{_python_buildid}-cherrypy
Requires:       python%{_python_buildid}-ipaddress
Requires:       python%{_python_buildid}-jwt
Requires:       python%{_python_buildid}-routes
Requires:       python%{_python_buildid}-werkzeug
%endif

in ceph.spec.in. so ceph-mgr-dashboard should depend on python-jwt.

but the funny thing is, this package was not installed:

$ grep ceph-mgr-dashboard /a//kchai-2019-12-11_19:03:22-rados:dashboard-master-distro-basic-mira/4593279/teuthology.log|grep Dependency
2019-12-11T19:25:27.273 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch
2019-12-11T19:25:27.382 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch
2019-12-11T19:25:28.734 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch
2019-12-11T19:25:28.843 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch
2019-12-11T19:30:06.543 INFO:teuthology.orchestra.run.mira088.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch
2019-12-11T19:30:06.654 INFO:teuthology.orchestra.run.mira100.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch

$ grep python-jwt /a//kchai-2019-12-11_19:03:22-rados:dashboard-master-distro-basic-mira/4593279/teuthology.log || echo bummer
bummer

i checked the dependency relationship of ceph-mgr-dashboard rpm package:

$ curl --silent --remote-name --location 'https://1.chacra.ceph.com/r/ceph/master/be58eb0a8d242bb4542177a1278be8ea1ba04045/centos/7/flavors/default/noarch/ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch.rpm'
$ rpm -qpR ceph-mgr-dashboard-15.0.0-8299.gbe58eb0.el7.noarch.rpm
/bin/sh
/bin/sh
ceph-grafana-dashboards = 2:15.0.0-8299.gbe58eb0.el7
ceph-mgr = 2:15.0.0-8299.gbe58eb0.el7
pyOpenSSL
python-cherrypy
python-ipaddress
python-jwt
python-routes
python-werkzeug
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(FileDigests) <= 4.6.0-1
rpmlib(PartialHardlinkSets) <= 4.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(PayloadIsXz) <= 5.2-1

it does depends on python-jwt. the crux of the problem is, as implied by the following backtrace

2019-12-11T19:29:39.399 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 214, in task
    module_suites.append(decorating_loader.loadTestsFromName(mod_name))
  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 84, in loadTestsFromName
    result = self._loadTestsFromName(name, module)
  File "/home/teuthworker/src/github.com_tchaikov_ceph_wip-43192/qa/tasks/cephfs_test_runner.py", line 57, in _loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_auth'

teuthology-worker runs on "teuthology" node, where it clones the qa suite repo. and this machine does not have python-jwt installed

kchai@teuthology:~$ python
Python 2.7.12 (default, Oct  8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import jwt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named jwt

$ dpkg -s python-jwt
dpkg-query: package 'python-jwt' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.

i am not able to query the log of apt

$ zgrep jwt /var/log/apt/*
gzip: /var/log/apt/term.log: Permission denied
gzip: /var/log/apt/term.log.10.gz: Permission denied
gzip: /var/log/apt/term.log.11.gz: Permission denied
gzip: /var/log/apt/term.log.12.gz: Permission denied
gzip: /var/log/apt/term.log.1.gz: Permission denied
gzip: /var/log/apt/term.log.2.gz: Permission denied
gzip: /var/log/apt/term.log.3.gz: Permission denied
gzip: /var/log/apt/term.log.4.gz: Permission denied
gzip: /var/log/apt/term.log.5.gz: Permission denied
gzip: /var/log/apt/term.log.6.gz: Permission denied
gzip: /var/log/apt/term.log.7.gz: Permission denied
gzip: /var/log/apt/term.log.8.gz: Permission denied
gzip: /var/log/apt/term.log.9.gz: Permission denied

i was wrong, teuthworker is using its own venv

ps aux |grep teuthworker| head -n1
teuthwo+  1385  1.0  0.2 191844 79616 ?        S    03:36   0:12 /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/python /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology -v --lock --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-12-11_23:13:54-multimds-wip-pdonnell-testing-20191211.205606-distro-basic-smithi/4593804 --name pdonnell-2019-12-11_23:13:54-multimds-wip-pdonnell-testing-20191211.205606-distro-basic-smithi --description multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_7.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} -- /tmp/teuthology-worker.wFUR2t.tmp

kchai@teuthology:~/teuthology$ source /home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/activate
(virtualenv) kchai@teuthology:~/teuthology$ python
Python 2.7.12 (default, Oct  8 2019, 14:14:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import jwt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named jwt
>>>

and teuthology/requirements.txt does not contain "PyJWT". so this is expected.

seems the right solution is to add "PyJWT" as an extra_require in setup.py,