Actions
Bug #52652
closedERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest) ... ERROR
2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.376 INFO:tasks.cephfs_test_runner:======================================================================
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.377 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/mgr/test_module_selftest.py", line 204, in test_module_commands
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner: self.wait_for_health_clear(timeout=30)
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 169, in wait_for_health_clear
2021-09-17T00:55:57.378 INFO:tasks.cephfs_test_runner: self.wait_until_true(is_clear, timeout)
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-09-17T00:55:57.379 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:Ran 6 tests in 439.008s
2021-09-17T00:55:57.380 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:
2021-09-17T00:55:57.381 INFO:tasks.cephfs_test_runner:======================================================================
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:ERROR: test_module_commands (tasks.mgr.test_module_selftest.TestModuleSelftest)
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-09-17T00:55:57.382 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/mgr/test_module_selftest.py", line 204, in test_module_commands
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner: self.wait_for_health_clear(timeout=30)
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 169, in wait_for_health_clear
2021-09-17T00:55:57.383 INFO:tasks.cephfs_test_runner: self.wait_until_true(is_clear, timeout)
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_8c021509ef8eb30460e69eb709f4458bcb47029c/qa/tasks/ceph_test_case.py", line 206, in wait_until_true
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s and {1} retries".format(elapsed, retry_count))
2021-09-17T00:55:57.384 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s and 0 retries
/a/yuriw-2021-09-16_18:23:18-rados-wip-yuri2-testing-2021-09-16-0923-distro-basic-smithi/6393748
Last related ticket https://tracker.ceph.com/issues/51743.
Updated by Neha Ojha over 2 years ago
/a/yuriw-2021-12-07_16:04:59-rados-wip-yuri5-testing-2021-12-06-1619-distro-default-smithi/6551099
Updated by Kamoltat (Junior) Sirivadhna over 2 years ago
/a/yuriw-2021-12-21_18:01:07-rados-wip-yuri3-testing-2021-12-21-0749-distro-default-smithi/6576526/
Updated by Sridhar Seshasayee over 2 years ago
/a/yuriw-2022-01-08_17:57:43-rados-wip-yuri8-testing-2022-01-07-1541-distro-default-smithi/6603271
Updated by Laura Flores about 2 years ago
This line is normal in the teuthology log, as this particular test invokes a synthetic exception. However...
2021-09-17T00:55:29.138 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'selftest' has failed: Synthetic exception in serve","count":
1},"muted":false}},"mutes":[]}
An extra line shows up in failed tests, indicating that a mgr module has crashed. This line is not invoked by the `test_module_commands` test. I believe this module crash is what's causing this test to fail (not deterministically) since it relies on a clear health message to pass.
2021-09-17T00:55:34.533 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"1 mgr modules have recently crashed","count":1},"muted":false}},"mutes":[]}
Updated by Laura Flores almost 2 years ago
/a/yuriw-2022-06-02_20:23:28-rados-wip-yuri-testing-2022-06-02-0810-quincy-distro-default-smithi/6860841
Updated by Kamoltat (Junior) Sirivadhna over 1 year ago
/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944203/
Updated by Laura Flores 11 months ago
- Status changed from In Progress to Closed
Haven't been seeing this.
Actions