Actions
Bug #47589
closedradosbench times out "reached maximum tries (800) after waiting for 4800 seconds"
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-09-22T15:54:14.509 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) // start . . 2020-09-22T16:25:11.776 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Total time run: 1857.16 2020-09-22T16:25:11.776 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Total writes made: 93280 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Write size: 65536 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Object size: 65536 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Bandwidth (MB/sec): 3.13921 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev Bandwidth: 12.4319 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max bandwidth (MB/sec): 80.1875 2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min bandwidth (MB/sec): 0 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Average IOPS: 50 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev IOPS: 198.91 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max IOPS: 1283 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min IOPS: 0 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Average Latency(s): 0.0549673 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev Latency(s): 5.81274 2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max latency(s): 1732.06 2020-09-22T16:25:11.779 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min latency(s): 0.00289317 2020-09-22T16:25:11.798 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Cleaning up (deleting benchmark objects) // start cleanup . . 2020-09-22T17:15:53.075 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/radosbench.py", line 140, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 464, in wait check_time() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (800) after waiting for 4800 seconds
The benchmark ran for 1857.16 seconds and cleanup did not finish in ~3042 secs, hence we timed out.
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench
/a/teuthology-2020-09-22_07:01:02-rados-master-distro-basic-smithi/5458599
Actions