Project

General

Profile

Bug #19540

.../rados/test.sh: line 9: 29593 Terminated (times out)

Added by Sage Weil over 2 years ago. Updated over 2 years ago.

Status:
Duplicate
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
Start date:
04/06/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

2017-04-06T21:06:51.961 INFO:tasks.workunit.client.0.smithi104.stderr:++ cleanup
2017-04-06T21:06:51.962 INFO:tasks.workunit.client.0.smithi104.stderr:++ pkill -P 29591
2017-04-06T21:06:51.970 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 9: 29593 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$f.log | sed \"s/^/$r: /\"" 
2017-04-06T21:06:51.970 INFO:tasks.workunit.client.0.smithi104.stderr:++ true
2017-04-06T21:06:51.971 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0...
2017-04-06T21:06:51.971 INFO:teuthology.orchestra.run.smithi104:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
2017-04-06T21:06:52.098 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-mgr-init/qa/tasks/workunit.py", line 450, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 414, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 149, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 171, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test rados/test.sh) on smithi104 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=880858be35b5b586cab1a9ca5605828c0c6de840 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

/a/sage-2017-04-06_17:19:30-rados:thrash-wip-mgr-init---basic-smithi/993669

I keep seeing these on my various runs and I can't find an actual failure in the test script.


Related issues

Duplicates Ceph - Bug #19430: objecter: full_try behavior not consistent with osd Resolved 03/30/2017

History

#1 Updated by Sage Weil over 2 years ago

/a/sage-2017-04-06_19:43:41-rados-wip-sage-testing---basic-smithi/994042

#2 Updated by Sage Weil over 2 years ago

  • Subject changed from .../rados/test.sh: line 9: 29593 Terminated to .../rados/test.sh: line 9: 29593 Terminated (times out)
  • Status changed from New to Verified

/a/sage-2017-04-07_04:46:46-rados:thrash-wip-bluestore-osr-drain-hang---basic-smithi/995352

it's api_aio just being slow

2017-04-07T06:28:22.210 INFO:tasks.workunit.client.0.smithi099.stderr:+ wait 16559
...3 hours...
2017-04-07T09:13:34.429 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio: Running main() from gmock_main.cc
...
2017-04-07T09:13:34.435 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio: [ RUN      ] LibRadosAio.FlushAsync
...2 minutes...
2017-04-07T09:15:11.877 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio: [       OK ] LibRadosAio.FlushAsync (3587 ms)
...
2017-04-07T09:15:11.885 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio:
...3 minutes...
2017-04-07T09:18:57.090 INFO:tasks.workunit.client.0.smithi099.stderr:++ cleanup
2017-04-07T09:18:57.090 INFO:tasks.workunit.client.0.smithi099.stderr:++ pkill -P 16557
2017-04-07T09:18:57.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 9: 16559 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$f.log | sed \"s/^/$r: /\"" 
2017-04-07T09:18:57.100 INFO:tasks.workunit.client.0.smithi099.stderr:++ true

and it times out.

#3 Updated by Sage Weil over 2 years ago

  • Status changed from Verified to Duplicate
2017-04-07T09:13:34.430 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio: [ RUN      ] LibRadosAio.PoolQuotaPP
2017-04-07T09:13:34.431 INFO:tasks.workunit.client.0.smithi099.stdout:                  api_aio: [       OK ] LibRadosAio.PoolQuotaPP (10377302 ms)

this is a side-effect of #19430.

#4 Updated by Sage Weil over 2 years ago

  • Duplicates Bug #19430: objecter: full_try behavior not consistent with osd added

Also available in: Atom PDF