Bug #50220
qa: dbench workload timeout
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-04-07T06:07:05.413 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2767653 14.00 MB/sec warmup 10736 sec latency 10735813.990 ms 2021-04-07T06:07:06.413 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2767834 14.00 MB/sec warmup 10737 sec latency 10736814.100 ms 2021-04-07T06:07:07.413 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2767999 14.00 MB/sec warmup 10738 sec latency 10737814.259 ms 2021-04-07T06:07:08.413 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768163 14.00 MB/sec warmup 10739 sec latency 10738814.337 ms 2021-04-07T06:07:09.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768333 14.00 MB/sec warmup 10740 sec latency 10739814.429 ms 2021-04-07T06:07:10.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768539 14.00 MB/sec warmup 10741 sec latency 10740814.497 ms 2021-04-07T06:07:11.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768671 14.00 MB/sec warmup 10742 sec latency 10741814.558 ms 2021-04-07T06:07:12.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768805 14.00 MB/sec warmup 10743 sec latency 10742814.658 ms 2021-04-07T06:07:13.415 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2768987 14.00 MB/sec warmup 10744 sec latency 10743814.739 ms 2021-04-07T06:07:14.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2769133 14.00 MB/sec warmup 10745 sec latency 10744814.826 ms 2021-04-07T06:07:15.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2769265 14.00 MB/sec warmup 10746 sec latency 10745814.902 ms 2021-04-07T06:07:16.414 INFO:tasks.workunit.client.0.smithi061.stdout: 9 2769412 14.00 MB/sec warmup 10747 sec latency 10746814.973 ms 2021-04-07T06:07:17.268 DEBUG:teuthology.orchestra.run:got remote process result: 124 2021-04-07T06:07:28.509 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:07:28.513 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:07:28.516 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:07:58.588 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:07:58.593 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:07:58.597 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:17.270 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: <Greenlet at 0x7fb6a3068348: copy_file_to(<paramiko.ChannelFile from <paramiko.Channel 218 (, <Logger tasks.workunit.client.0.smithi061.stderr (, None, False)> 2021-04-07T06:08:28.715 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:28.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:28.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:58.841 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:58.851 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:08:58.855 DEBUG:teuthology.orchestra.run.smithi170:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2021-04-07T06:09:17.273 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: <Greenlet at 0x7fb6a3068048: copy_file_to(<paramiko.ChannelFile from <paramiko.Channel 218 (, <Logger tasks.workunit.client.0.smithi061.stdout (, None, False)> 2021-04-07T06:09:17.308 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2021-04-07T06:09:17.309 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2021-04-07T06:09:17.665 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e4f7126b95459fb0239be660756fb943526cec3f/qa/tasks/workunit.py", line 147, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_e4f7126b95459fb0239be660756fb943526cec3f/qa/tasks/workunit.py", line 297, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e4f7126b95459fb0239be660756fb943526cec3f/qa/tasks/workunit.py", line 425, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 183, in _raise_for_status node=self.hostname, label=self.label teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi061 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e4f7126b95459fb0239be660756fb943526cec3f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
From: /ceph/teuthology-archive/pdonnell-2021-04-07_02:12:41-fs-wip-pdonnell-testing-20210406.213012-distro-basic-smithi/6025533/teuthology.log
Doesn't look stuck. Just slow but I don't see a particular reason why.
History
#1 Updated by Patrick Donnelly almost 2 years ago
/ceph/teuthology-archive/pdonnell-2021-04-15_01:35:57-fs-wip-pdonnell-testing-20210414.230315-distro-basic-smithi/6047657/teuthology.log
testing kernel
#2 Updated by Patrick Donnelly 7 months ago
- Target version deleted (
v17.0.0)