Project

General

Profile

Actions
Copy link

Bug #44347

closed

mgr/dashboard: nightly dashboard backend API test failure "test_standby (tasks.mgr.test_dashboard.TestDashboard)"

Added by Laura Paduano about 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Mykola Golub
Category:
Testing & QA
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0
ceph-qa-suite:
Pull request ID:
33624
Crash signature (v1):
Crash signature (v2):

Description

2020-02-28 03:18:01,653.653 INFO:tasks.mgr.mgr_test_case:Loading Mgr module dashboard ...
2020-02-28 03:18:01,654.654 INFO:__main__:Running ['./bin/ceph', 'mgr', 'module', 'enable', 'dashboard', '--force']
2020-02-28 03:18:02,874.874 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:08,298.298 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:08,786.786 INFO:tasks.mgr.mgr_test_case:Restarted after module load (new active y/4632)
2020-02-28 03:18:08,789.789 INFO:__main__:Running ['./bin/ceph', 'dashboard', 'create-self-signed-cert']
2020-02-28 03:18:09,303.303 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:09,725.725 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:10,178.178 INFO:tasks.mgr.mgr_test_case:Found dashboard at https://bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9:7789/ (daemon y/4632)
2020-02-28 03:18:10,179.179 INFO:tasks.mgr.test_dashboard:Originally running manager 'y' at https://bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9:7789/
2020-02-28 03:18:10,179.179 INFO:__main__:Running ['./bin/ceph', 'mgr', 'fail', u'y']
2020-02-28 03:18:11,438.438 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:16,875.875 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:17,359.359 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:17,827.827 INFO:__main__:Running ['./bin/ceph', 'mgr', 'dump', '--format=json-pretty']
2020-02-28 03:18:18,312.312 INFO:tasks.mgr.mgr_test_case:Found dashboard at https://bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9:7790/ (daemon x/4626)
2020-02-28 03:18:18,313.313 INFO:tasks.mgr.test_dashboard:After failover running manager 'x' at https://bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9:7790/
2020-02-28 03:18:18,326.326 INFO:__main__:test_standby (tasks.mgr.test_dashboard.TestDashboard) ... ERROR
2020-02-28 03:18:18,326.326 INFO:__main__:Running ['./bin/ceph', 'config', 'set', 'mgr', 'mgr/dashboard/standby_behaviour', 'redirect']
2020-02-28 03:18:18,802.802 INFO:__main__:Running ['./bin/ceph', 'config', 'set', 'mgr', 'mgr/dashboard/standby_error_status_code', '500']
2020-02-28 03:18:19,252.252 INFO:__main__:Stopped test: test_standby (tasks.mgr.test_dashboard.TestDashboard) in 30.638948s
2020-02-28 03:18:19,253.253 INFO:__main__:
2020-02-28 03:18:19,253.253 INFO:__main__:======================================================================
2020-02-28 03:18:19,253.253 INFO:__main__:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2020-02-28 03:18:19,254.254 INFO:__main__:----------------------------------------------------------------------
2020-02-28 03:18:19,254.254 INFO:__main__:Traceback (most recent call last):
2020-02-28 03:18:19,254.254 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/test_dashboard.py", line 53, in test_standby
2020-02-28 03:18:19,254.254 INFO:__main__:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2020-02-28 03:18:19,254.254 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/api.py", line 75, in get
2020-02-28 03:18:19,254.254 INFO:__main__:    return request('get', url, params=params, **kwargs)
2020-02-28 03:18:19,254.254 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/api.py", line 60, in request
2020-02-28 03:18:19,254.254 INFO:__main__:    return session.request(method=method, url=url, **kwargs)
2020-02-28 03:18:19,255.255 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request
2020-02-28 03:18:19,255.255 INFO:__main__:    resp = self.send(prep, **send_kwargs)
2020-02-28 03:18:19,255.255 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send
2020-02-28 03:18:19,255.255 INFO:__main__:    r = adapter.send(request, **kwargs)
2020-02-28 03:18:19,255.255 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
2020-02-28 03:18:19,255.255 INFO:__main__:    raise ConnectionError(e, request=request)
2020-02-28 03:18:19,255.255 INFO:__main__:ConnectionError: HTTPSConnectionPool(host='bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9', port=7789): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f04f468e8d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-02-28 03:18:19,256.256 INFO:__main__:
2020-02-28 03:18:19,256.256 INFO:__main__:----------------------------------------------------------------------
2020-02-28 03:18:19,256.256 INFO:__main__:Ran 1 test in 43.031s
2020-02-28 03:18:19,256.256 INFO:__main__:
2020-02-28 03:18:19,256.256 INFO:__main__:FAILED (errors=1)
2020-02-28 03:18:19,256.256 INFO:__main__:
2020-02-28 03:18:19,256.256 INFO:__main__:======================================================================
2020-02-28 03:18:19,257.257 INFO:__main__:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2020-02-28 03:18:19,257.257 INFO:__main__:----------------------------------------------------------------------
2020-02-28 03:18:19,257.257 INFO:__main__:Traceback (most recent call last):
2020-02-28 03:18:19,257.257 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa/tasks/mgr/test_dashboard.py", line 53, in test_standby
2020-02-28 03:18:19,257.257 INFO:__main__:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2020-02-28 03:18:19,257.257 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/api.py", line 75, in get
2020-02-28 03:18:19,257.257 INFO:__main__:    return request('get', url, params=params, **kwargs)
2020-02-28 03:18:19,257.257 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/api.py", line 60, in request
2020-02-28 03:18:19,258.258 INFO:__main__:    return session.request(method=method, url=url, **kwargs)
2020-02-28 03:18:19,258.258 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 524, in request
2020-02-28 03:18:19,258.258 INFO:__main__:    resp = self.send(prep, **send_kwargs)
2020-02-28 03:18:19,258.258 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 637, in send
2020-02-28 03:18:19,258.258 INFO:__main__:    r = adapter.send(request, **kwargs)
2020-02-28 03:18:19,258.258 INFO:__main__:  File "/tmp/tmp.zOs4dCijRO/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
2020-02-28 03:18:19,258.258 INFO:__main__:    raise ConnectionError(e, request=request)
2020-02-28 03:18:19,258.258 INFO:__main__:ConnectionError: HTTPSConnectionPool(host='bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9', port=7789): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f04f468e8d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-02-28 03:18:19,259.259 INFO:__main__:
Using guessed paths /home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/build/lib/ ['/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/qa', '/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/build/lib/cython_modules/lib.3', '/home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/src/pybind']

Displaying last 60000 lines of: /home/jenkins-build/build/workspace/ceph-api-nightly-master-backend/src/pybind/mgr/dashboard/../../../../build/out/mgr.x.log
Actions
Copy link
 #1

Updated by Laura Paduano about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Laura Paduano about 4 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Alfonso Martínez about 4 years ago 

2020-02-28T11:51:09.522+0000 7f2af1f73ec0  4 mgr[py] load_subclass_of: found class: 'prometheus.StandbyModule'
2020-02-28T11:51:09.522+0000 7f2af1f73ec0  4 mgr[py] Standby mode available in module 'prometheus'
2020-02-28T11:51:09.522+0000 7f2af1f73ec0  1 mgr[py] Loading python module 'rbd_support'
2020-02-28T11:51:09.542+0000 7f2af1f73ec0 10 mgr[py] Computed sys.path '/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/dashboard/../../../../qa:/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/dashboard/../../../../build/lib/cython_modules/lib.3/:/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/dashboard/../../../../build/src/pybind:/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/dashboard/../../../../src/python-common:/tmp/tmp.HZt6vxfNcC/coverage-venv/lib/python3.6/site-packages:/usr/lib/python36.zip:/usr/lib/python3.6:/usr/lib/python3.6:/usr/lib/python3.6/lib-dynload:/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr:/usr/lib/python3.6/site-packages:/usr/local/lib/python3.6/dist-packages:/usr/lib/python3/dist-packages:/usr/lib/python3.6/dist-packages'
2020-02-28T11:51:09.562+0000 7f2af1f73ec0 -1 mgr[py] Module not found: 'rbd_support'
2020-02-28T11:51:09.562+0000 7f2af1f73ec0 -1 mgr[py] Traceback (most recent call last):
  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/rbd_support/__init__.py", line 1, in <module>
    from .module import Module
  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/rbd_support/module.py", line 13, in <module>
    from .mirror_snapshot_schedule import MirrorSnapshotScheduleHandler
  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/rbd_support/mirror_snapshot_schedule.py", line 12, in <module>
    from .schedule import LevelSpec, Interval, StartTime, Schedule, Schedules
  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/src/pybind/mgr/rbd_support/schedule.py", line 7, in <module>
    from dateutil.parser import parse
ModuleNotFoundError: No module named 'dateutil'
Actions
Copy link
 #4

Updated by Mykola Golub about 4 years ago

I don't think the rbd_support module failure due to not found dateutil is related to this issue. Because it happens only on jenkins, while the original issue was reported for both jenkins and teuthology. Anyway, the dateutil module dependency issue should be fixed by this PR: https://github.com/ceph/ceph/pull/33624

Actions
Copy link
 #5

Updated by Nathan Cutler about 4 years ago

The test is failing because it can't reach a URL: ConnectionError: HTTPSConnectionPool(host='bionic-huge--75220935-8ef1-4b56-91e9-37a8a71cd7e9', port=7789): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f04f468e8d0>: Failed to establish a new connection: [Errno 111] Connection refused',)

There is a known Internet connectivity issue in the Sepia lab these days: #44343

Maybe the problem will go away when that is fixed.

Actions
Copy link
 #6

Updated by Nathan Cutler about 4 years ago

Nevermind, this has nothing to do with #44343

The failure in the teuthology run http://qa-proxy.ceph.com/teuthology/laura-2020-02-28_12:00:24-rados:dashboard-wip-laura-testing-33178-distro-basic-smithi/4808886/teuthology.log happened because of this:

2020-02-28T12:24:02.692 INFO:tasks.mgr.mgr_test_case:Found dashboard at https://smithi065.front.sepia.ceph.com:7791/ (daemon z/4878)
2020-02-28T12:24:02.693 INFO:tasks.mgr.test_dashboard:After failover running manager 'z' at https://smithi065.front.sepia.ceph.com:7791/
2020-02-28T12:24:02.764 INFO:tasks.cephfs_test_runner:test_standby (tasks.mgr.test_dashboard.TestDashboard) ... ERROR

Which triggered

2020-02-28T12:24:02.765 INFO:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early config set mgr mgr/dashboard/standby_behaviour redirect

and when that finishes we see the test fail:

2020-02-28T12:24:03.338 INFO:tasks.cephfs_test_runner:======================================================================
2020-02-28T12:24:03.338 INFO:tasks.cephfs_test_runner:ERROR: test_standby (tasks.mgr.test_dashboard.TestDashboard)
2020-02-28T12:24:03.339 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-02-28T12:24:03.339 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-02-28T12:24:03.339 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-33178/qa/tasks/mgr/test_dashboard.py", line 53, in test_standby
2020-02-28T12:24:03.339 INFO:tasks.cephfs_test_runner:    r = requests.get(original_uri, allow_redirects=False, verify=False)
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 75, in get
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:    return request('get', url, params=params, **kwargs)
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/api.py", line 60, in request
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:    return session.request(method=method, url=url, **kwargs)
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:    resp = self.send(prep, **send_kwargs)
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
2020-02-28T12:24:03.340 INFO:tasks.cephfs_test_runner:    r = adapter.send(request, **kwargs)
2020-02-28T12:24:03.341 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
2020-02-28T12:24:03.341 INFO:tasks.cephfs_test_runner:    raise ConnectionError(e, request=request)
2020-02-28T12:24:03.341 INFO:tasks.cephfs_test_runner:ConnectionError: HTTPSConnectionPool(host='smithi116.front.sepia.ceph.com', port=7789): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1cc5c121d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
2020-02-28T12:24:03.341 INFO:tasks.cephfs_test_runner:

It fails because it's trying to communicate with the pre-failover dashboard on https://smithi116.front.sepia.ceph.com:7789/

but at this point the dashboard is running on a different URL: https://smithi065.front.sepia.ceph.com:7791/

Actions
Copy link
 #7

Updated by Laura Paduano about 4 years ago

  • Status changed from New to Resolved
  • Assignee set to Mykola Golub
  • Pull request ID set to 33624
Actions
Copy link
 #8

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 151 to Testing & QA
Actions
Copy link

Also available in: Atom PDF