Project

General

Profile

Bug #42837

qa: test_ops_throttle failed with `RuntimeError: Ops in flight high water is unexpectedly low (0 / 16)`

Added by Ramana Raja over 4 years ago. Updated about 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:======================================================================
2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:ERROR: test_ops_throttle (tasks.cephfs.test_strays.TestStrays)
2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-11-05T21:07:03.545 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-11-05T21:07:03.546 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 26, in test_ops_throttle
2019-11-05T21:07:03.546 INFO:tasks.cephfs_test_runner:    self._test_throttling(self.OPS_THROTTLE)
2019-11-05T21:07:03.547 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 98, in _test_throttling
2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:    return self._do_test_throttling(throttle_type)
2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 246, in _do_test_throttling
2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:    ops_high_water, mds_max_purge_ops
2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:RuntimeError: Ops in flight high water is unexpectedly low (0 / 16)
2019-11-05T21:07:03.548 INFO:tasks.cephfs_test_runner:
2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:Ran 10 tests in 1128.908s
2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:
2019-11-05T21:07:03.549 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner:
2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner:======================================================================
2019-11-05T21:07:03.550 INFO:tasks.cephfs_test_runner:ERROR: test_ops_throttle (tasks.cephfs.test_strays.TestStrays)
2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-11-05T21:07:03.551 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 26, in test_ops_throttle
2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner:    self._test_throttling(self.OPS_THROTTLE)
2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 98, in _test_throttling
2019-11-05T21:07:03.555 INFO:tasks.cephfs_test_runner:    return self._do_test_throttling(throttle_type)
2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191105.085703/qa/tasks/cephfs/test_strays.py", line 246, in _do_test_throttling
2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner:    ops_high_water, mds_max_purge_ops
2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner:RuntimeError: Ops in flight high water is unexpectedly low (0 / 16)
2019-11-05T21:07:03.556 INFO:tasks.cephfs_test_runner:
2019-11-05T21:07:03.557 ERROR:teuthology.run_tasks:Saw exception from tasks.

See this in Patrick's recent test run based on master and in Yuri's nautilus run,

In Patrick's run based on master, two jobs have the same failure
http://qa-proxy.ceph.com/teuthology/pdonnell-2019-11-05_14:35:44-fs-wip-pdonnell-testing-20191105.085703-distro-basic-smithi/4474675/teuthology.log
http://qa-proxy.ceph.com/teuthology/pdonnell-2019-11-05_14:35:44-fs-wip-pdonnell-testing-20191105.085703-distro-basic-smithi/4474838/teuthology.log

In Yuri's nautilus run, see the same failure in a job
http://qa-proxy.ceph.com/teuthology/yuriw-2019-11-13_20:47:25-fs-wip-yuri7-testing-2019-11-13-1707-nautilus-testing-basic-smithi/4505412/teuthology.log

History

#1 Updated by Venky Shankar over 4 years ago

might be same as: https://tracker.ceph.com/issues/16881

but the test case is different, so not marking as dup.

#2 Updated by Patrick Donnelly over 4 years ago

  • Project changed from Ceph to CephFS

#3 Updated by Patrick Donnelly about 4 years ago

  • Target version deleted (v15.0.0)

Also available in: Atom PDF