Ceph » RADOS

2020-09-22T15:54:14.509 INFO:tasks.radosbench:joining radosbench (timing out after 4800s) // start
.
.
2020-09-22T16:25:11.776 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Total time run:         1857.16
2020-09-22T16:25:11.776 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Total writes made:      93280
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Write size:             65536
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Object size:            65536
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Bandwidth (MB/sec):     3.13921
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev Bandwidth:       12.4319
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max bandwidth (MB/sec): 80.1875
2020-09-22T16:25:11.777 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min bandwidth (MB/sec): 0
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Average IOPS:           50
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev IOPS:            198.91
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max IOPS:               1283
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min IOPS:               0
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Average Latency(s):     0.0549673
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Stddev Latency(s):      5.81274
2020-09-22T16:25:11.778 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Max latency(s):         1732.06
2020-09-22T16:25:11.779 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Min latency(s):         0.00289317
2020-09-22T16:25:11.798 INFO:tasks.radosbench.radosbench.0.smithi039.stdout:Cleaning up (deleting benchmark objects) // start cleanup
.
.
2020-09-22T17:15:53.075 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 69, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/full_sequential.py", line 37, in task
    mgr.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/radosbench.py", line 140, in task
    run.wait(radosbench.values(), timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 464, in wait
    check_time()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 133, in __call__
    raise MaxWhileTries(error_msg)
teuthology.exceptions.MaxWhileTries: reached maximum tries (800) after waiting for 4800 seconds

The benchmark ran for 1857.16 seconds and cleanup did not finish in ~3042 secs, hence we timed out.

rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench

/a/teuthology-2020-09-22_07:01:02-rados-master-distro-basic-smithi/5458599