Project

General

Profile

Actions

Bug #16690

closed

'cecephtoolphtool' timeout in rados run

Added by Yuri Weinstein almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/yuriw-2016-07-12_13:07:42-rados-wip-yuri-testing_2016_7_12-distro-basic-smithi/
Job: 309951, 310203
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-07-12_13:07:42-rados-wip-yuri-testing_2016_7_12-distro-basic-smithi/309951/teuthology.log

2016-07-12T22:20:56.615 INFO:tasks.workunit.client.0.smithi045.stderr:grep: /CEPH_WATCH_17662: No such file or directory
2016-07-12T22:20:56.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:175: ceph_watch_wait:  sleep 1
2016-07-12T22:20:57.617 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:178: ceph_watch_wait:  kill 23753
2016-07-12T22:20:57.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh: line 178: kill: (23753) - No such process
2016-07-12T22:20:57.618 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:1: ceph_watch_wait:  rm -fr cephtool.9E3
2016-07-12T22:20:57.620 INFO:tasks.workunit:Stopping ['cephtool'] on client.0...
2016-07-12T22:20:57.621 INFO:teuthology.orchestra.run.smithi045:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/workunit.client.0 /home/ubuntu/cephtest/clone'
2016-07-12T22:20:57.699 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 404, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 394, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 121, in wait
    label=self.label)
CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aec9745e170e947a5eed80f5b844c2e05ce183aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'
2016-07-12T22:20:57.708 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 66, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 45, in run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 125, in task
    config.get('subdir'), timeout=timeout)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 273, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 404, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 394, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 121, in wait
    label=self.label)
CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aec9745e170e947a5eed80f5b844c2e05ce183aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'
2016-07-12T22:20:58.065 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=ffcfc6b8d0634408936e86c81329eacf
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 66, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 45, in run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 125, in task
    config.get('subdir'), timeout=timeout)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 273, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 404, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 394, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 121, in wait
    label=self.label)
CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=aec9745e170e947a5eed80f5b844c2e05ce183aa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'
Actions #1

Updated by Yuri Weinstein almost 8 years ago

  • Description updated (diff)
Actions #2

Updated by Kefu Chai almost 8 years ago

  • Status changed from New to Resolved
  • Assignee set to Kefu Chai
2016-07-12T22:20:56.615 INFO:tasks.workunit.client.0.smithi045.stderr:grep: /CEPH_WATCH_17662: No such file or directory

already fixed by https://github.com/ceph/ceph/pull/10306

Actions

Also available in: Atom PDF