Bug #42837
qa: test_ops_throttle failed with `RuntimeError: Ops in flight high water is unexpectedly low (0 / 16)`
0%
Description
2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:====================================================================== 2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:ERROR: test_ops_throttle (tasks.cephfs.test_strays.TestStrays) 2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2019-11-05T21:07:03.546 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 26, in test_ops_throttle 2019-11-05T21:07:03.546 INFO:tasks.cephfs_test_runner: self._test_throttling(self.OPS_THROTTLE) 2019-11-05T21:07:03.547 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 98, in _test_throttling 2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner: return self._do_test_throttling(throttle_type) 2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 246, in _do_test_throttling 2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner: ops_high_water, mds_max_purge_ops 2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:RuntimeError: Ops in flight high water is unexpectedly low (0 / 16) 2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner: 2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:Ran 10 tests in 1128.908s 2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner: 2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner: 2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner:====================================================================== 2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner:ERROR: test_ops_throttle (tasks.cephfs.test_strays.TestStrays) 2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 26, in test_ops_throttle 2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner: self._test_throttling(self.OPS_THROTTLE) 2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 98, in _test_throttling 2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner: return self._do_test_throttling(throttle_type) 2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 246, in _do_test_throttling 2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner: ops_high_water, mds_max_purge_ops 2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner:RuntimeError: Ops in flight high water is unexpectedly low (0 / 16) 2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner: 2019-11-05T21:07:03.557 ERROR:teuthology.run_tasks:Saw exception from tasks.
See this in Patrick's recent test run based on master and in Yuri's nautilus run,
In Patrick's run based on master, two jobs have the same failure
http://qa-proxy.ceph.com/teuthology/pdonnell-2019-11-05_14:35:44-fs-wip-pdonnell-testing-20191105.085703-distro-basic-smithi/4474675/teuthology.log
http://qa-proxy.ceph.com/teuthology/pdonnell-2019-11-05_14:35:44-fs-wip-pdonnell-testing-20191105.085703-distro-basic-smithi/4474838/teuthology.log
In Yuri's nautilus run, see the same failure in a job
http://qa-proxy.ceph.com/teuthology/yuriw-2019-11-13_20:47:25-fs-wip-yuri7-testing-2019-11-13-1707-nautilus-testing-basic-smithi/4505412/teuthology.log
History
#1 Updated by Venky Shankar over 4 years ago
might be same as: https://tracker.ceph.com/issues/16881
but the test case is different, so not marking as dup.
#2 Updated by Patrick Donnelly over 4 years ago
- Project changed from Ceph to CephFS
#3 Updated by Patrick Donnelly about 4 years ago
- Target version deleted (
v15.0.0)