Project

General

Profile

Actions
Copy link

Bug #45963

closed

mgr/dashboard: ceph dashboard backend API test ERROR: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)

Added by Laura Paduano almost 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sebastian Wagner
Category:
Testing & QA
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0
ceph-qa-suite:
Pull request ID:
35521
Crash signature (v1):
Crash signature (v2):

Description

Seen the following error on Jenkins jobs "ceph-api-nightly-master-backend" and "ceph-dashboard-pr-backend":

2020-06-10 01:32:20,579.579 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "9702c4d4-3bc5-46e6-83c1-ec764882e102"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 01:32:20,580.580 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_disabled_module']
2020-06-10 01:32:21,524.524 INFO:__main__:test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) ... ERROR
2020-06-10 01:32:21,525.525 INFO:__main__:Stopped test: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) in 34.241966s
2020-06-10 01:32:21,525.525 INFO:__main__:
2020-06-10 01:32:21,525.525 INFO:__main__:======================================================================
2020-06-10 01:32:21,526.526 INFO:__main__:ERROR: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2020-06-10 01:32:21,526.526 INFO:__main__:----------------------------------------------------------------------
2020-06-10 01:32:21,526.526 INFO:__main__:Traceback (most recent call last):
2020-06-10 01:32:21,526.526 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_mgr_module.py", line 36, in test_list_disabled_module
2020-06-10 01:32:21,526.526 INFO:__main__:    self.wait_until_rest_api_accessible()
2020-06-10 01:32:21,526.526 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_mgr_module.py", line 30, in wait_until_rest_api_accessible
2020-06-10 01:32:21,526.526 INFO:__main__:    self.wait_until_true(_check_connection, timeout=30)
2020-06-10 01:32:21,527.527 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/ceph_test_case.py", line 194, in wait_until_true
2020-06-10 01:32:21,527.527 INFO:__main__:    raise RuntimeError("Timed out after {0}s".format(elapsed))
2020-06-10 01:32:21,527.527 INFO:__main__:RuntimeError: Timed out after 30s
2020-06-10 01:32:21,527.527 INFO:__main__:
2020-06-10 01:32:21,527.527 INFO:__main__:----------------------------------------------------------------------
2020-06-10 01:32:21,527.527 INFO:__main__:Ran 81 tests in 1058.910s
2020-06-10 01:32:21,527.527 INFO:__main__:
2020-06-10 01:32:21,527.527 INFO:__main__:FAILED (errors=1)
2020-06-10 01:32:21,528.528 INFO:__main__:
2020-06-10 01:32:21,528.528 INFO:__main__:======================================================================
2020-06-10 01:32:21,528.528 INFO:__main__:ERROR: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2020-06-10 01:32:21,528.528 INFO:__main__:----------------------------------------------------------------------
2020-06-10 01:32:21,528.528 INFO:__main__:Traceback (most recent call last):
2020-06-10 01:32:21,528.528 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_mgr_module.py", line 36, in test_list_disabled_module
2020-06-10 01:32:21,528.528 INFO:__main__:    self.wait_until_rest_api_accessible()
2020-06-10 01:32:21,528.528 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_mgr_module.py", line 30, in wait_until_rest_api_accessible
2020-06-10 01:32:21,528.528 INFO:__main__:    self.wait_until_true(_check_connection, timeout=30)
2020-06-10 01:32:21,529.529 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/ceph_test_case.py", line 194, in wait_until_true
2020-06-10 01:32:21,529.529 INFO:__main__:    raise RuntimeError("Timed out after {0}s".format(elapsed))
2020-06-10 01:32:21,529.529 INFO:__main__:RuntimeError: Timed out after 30s

- https://jenkins.ceph.com/view/mgr-dashboard/job/ceph-api-nightly-master-backend/193/
- https://jenkins.ceph.com/view/mgr-dashboard/job/ceph-dashboard-pr-backend/1307
- https://jenkins.ceph.com/view/mgr-dashboard/job/ceph-dashboard-pr-backend/1310/

Files

Screenshot_20200610_110642.png (46 KB) Screenshot_20200610_110642.png Laura Paduano, 06/10/2020 09:18 AM
Actions
Copy link
 #1

Updated by Laura Paduano almost 4 years ago

  • Description updated (diff)

Can also reproduce it in my local ceph-dev-docker environment when running the run-backend-api-test.sh:

2020-06-10 08:03:57,424.424 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_get']
2020-06-10 08:03:58,811.811 INFO:__main__:test_get (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) ... ok
2020-06-10 08:03:58,812.812 INFO:__main__:Stopped test: test_get (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) in 3.657736s
2020-06-10 08:03:58,812.812 INFO:__main__:Starting test: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2020-06-10 08:03:58,812.812 INFO:__main__:Running ['./bin/ceph', 'log', 'Starting test tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_disabled_module']
2020-06-10 08:03:59,825.825 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-06-10 08:04:00,480.480 INFO:__main__:Running ['./bin/ceph', 'mgr', 'module', 'disable', 'iostat']
2020-06-10 08:04:01,890.890 INFO:tasks.mgr.dashboard.helper:command result: 
2020-06-10 08:04:01,891.891 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:01,911.911 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "7ae50015-340f-4027-adb2-d9c1a4f1280a"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:06,916.916 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:06,932.932 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "c8576e01-1737-48c2-a6bd-ebc37c0fd4ee"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:11,937.937 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:11,953.953 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "bc8333f1-5368-4dab-a115-3893630b34fc"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:16,954.954 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:16,969.969 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "5d4b4047-8351-4c15-b66f-a3c0a5b0684e"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:21,974.974 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:21,989.989 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "dc05b1e9-2b5a-44d2-98a3-6dc94210eb3d"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:26,995.995 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:27,010.010 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "f2c2017e-c1da-4904-a723-a746a292d25c"}                                                                                                                                                                                                                                                                                                                        
2020-06-10 08:04:32,016.016 INFO:tasks.mgr.dashboard.helper:Request GET to https://localhost:7789/api/mgr/module
/tmp/tmp.fwe7xEGy0R/venv/lib/python3.8/site-packages/urllib3/connectionpool.py:842: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
2020-06-10 08:04:32,031.031 ERROR:tasks.mgr.dashboard.helper:Request response: {"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "bef03c14-51cb-4259-881f-20ddf2957285"}
2020-06-10 08:04:32,031.031 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_disabled_module']
2020-06-10 08:04:32,779.779 INFO:__main__:test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) ... ERROR
2020-06-10 08:04:32,779.779 INFO:__main__:Stopped test: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) in 33.967129s
2020-06-10 08:04:32,780.780 INFO:__main__:
2020-06-10 08:04:32,780.780 INFO:__main__:======================================================================
2020-06-10 08:04:32,780.780 INFO:__main__:ERROR: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2020-06-10 08:04:32,781.781 INFO:__main__:----------------------------------------------------------------------
2020-06-10 08:04:32,781.781 INFO:__main__:Traceback (most recent call last):
2020-06-10 08:04:32,781.781 INFO:__main__:  File "/ceph/qa/tasks/mgr/dashboard/test_mgr_module.py", line 36, in test_list_disabled_module
2020-06-10 08:04:32,781.781 INFO:__main__:    self.wait_until_rest_api_accessible()
2020-06-10 08:04:32,781.781 INFO:__main__:  File "/ceph/qa/tasks/mgr/dashboard/test_mgr_module.py", line 30, in wait_until_rest_api_accessible
2020-06-10 08:04:32,781.781 INFO:__main__:    self.wait_until_true(_check_connection, timeout=30)
2020-06-10 08:04:32,782.782 INFO:__main__:  File "/ceph/qa/tasks/ceph_test_case.py", line 194, in wait_until_true
2020-06-10 08:04:32,782.782 INFO:__main__:    raise RuntimeError("Timed out after {0}s".format(elapsed))
2020-06-10 08:04:32,782.782 INFO:__main__:RuntimeError: Timed out after 30s
2020-06-10 08:04:32,782.782 INFO:__main__:
2020-06-10 08:04:32,782.782 INFO:__main__:----------------------------------------------------------------------
2020-06-10 08:04:32,782.782 INFO:__main__:Ran 81 tests in 1227.530s
2020-06-10 08:04:32,783.783 INFO:__main__:
2020-06-10 08:04:32,783.783 INFO:__main__:FAILED (errors=1)
2020-06-10 08:04:32,783.783 INFO:__main__:
2020-06-10 08:04:32,783.783 INFO:__main__:======================================================================
2020-06-10 08:04:32,783.783 INFO:__main__:ERROR: test_list_disabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
2020-06-10 08:04:32,783.783 INFO:__main__:----------------------------------------------------------------------
2020-06-10 08:04:32,784.784 INFO:__main__:Traceback (most recent call last):
2020-06-10 08:04:32,784.784 INFO:__main__:  File "/ceph/qa/tasks/mgr/dashboard/test_mgr_module.py", line 36, in test_list_disabled_module
2020-06-10 08:04:32,784.784 INFO:__main__:    self.wait_until_rest_api_accessible()
2020-06-10 08:04:32,784.784 INFO:__main__:  File "/ceph/qa/tasks/mgr/dashboard/test_mgr_module.py", line 30, in wait_until_rest_api_accessible
2020-06-10 08:04:32,784.784 INFO:__main__:    self.wait_until_true(_check_connection, timeout=30)
2020-06-10 08:04:32,784.784 INFO:__main__:  File "/ceph/qa/tasks/ceph_test_case.py", line 194, in wait_until_true
2020-06-10 08:04:32,785.785 INFO:__main__:    raise RuntimeError("Timed out after {0}s".format(elapsed))
2020-06-10 08:04:32,785.785 INFO:__main__:RuntimeError: Timed out after 30s
2020-06-10 08:04:32,785.785 INFO:__main__:

Actions
Copy link
 #2

Updated by Laura Paduano almost 4 years ago

Did not face this error in the QA (suite rados/dashboard) run that I've scheduled yesterday (http://pulpito.ceph.com/laura-2020-06-08_10:28:37-rados:dashboard-wip-laura-testing-35007-35379-distro-basic-smithi/)

Actions
Copy link
 #3

Updated by Laura Paduano almost 4 years ago

Going to Cluster >> Manager modules in the UI:

Actions
Copy link
 #4

Updated by Kiefer Chang almost 4 years ago

Add the traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/usr/lib/python3.8/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/usr/lib/python3.8/site-packages/cherrypy/lib/encoding.py", line 219, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/cherrypy/_cptools.py", line 230, in wrap
    return self.newhandler(innerfunc, *args, **kwargs)
  File "/ceph/src/pybind/mgr/dashboard/services/exception.py", line 94, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/ceph/src/pybind/mgr/dashboard/controllers/__init__.py", line 665, in inner
    ret = func(*args, **kwargs)
  File "/ceph/src/pybind/mgr/dashboard/controllers/__init__.py", line 860, in wrapper
    return func(*vpath, **params)
  File "/ceph/src/pybind/mgr/dashboard/controllers/mgr_modules.py", line 34, in list
    'options': self._convert_module_options(
  File "/ceph/src/pybind/mgr/dashboard/controllers/mgr_modules.py", line 168, in _convert_module_options
    option[name] = int(option[name])
ValueError: invalid literal for int() with base 10: 'None'

Adding some debug message and it seems to fail at the `migration_current` option introduced in this PR

Actions
Copy link
 #5

Updated by Lenz Grimmer almost 4 years ago

  • Regression changed from No to Yes
Actions
Copy link
 #6

Updated by Volker Theile almost 4 years ago

IMO the problem is not the Dashboard code. If the type of 'migration_current' is configured as 'int', then the code can assume the given value can be converted from 'string' to 'int'. There is no need to check for 'None' IMO. The default value of the PR that introduces the regression must be fixed instead.

Actions
Copy link
 #7

Updated by Laura Paduano almost 4 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #8

Updated by Sebastian Wagner almost 4 years ago

  • Status changed from New to In Progress
  • Assignee set to Sebastian Wagner
Actions
Copy link
 #9

Updated by Sebastian Wagner almost 4 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 35521
Actions
Copy link
 #11

Updated by Kefu Chai almost 4 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #12

Updated by Kefu Chai almost 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link
 #13

Updated by Ernesto Puerta about 3 years ago

  • Project changed from mgr to Dashboard
  • Category changed from 151 to Testing & QA
Actions
Copy link

Also available in: Atom PDF