Bug #58757: qa: Command failed (workunit test suites/fsstress.sh) - CephFS - Ceph

Actions

Copy link

Bug #58757

closed

qa: Command failed (workunit test suites/fsstress.sh)

Added by Jos Collin about 1 year ago. Updated about 1 year ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Jos Collin

Category:

Target version:

% Done:

Source:

Tags:

quincy

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Labels (FS):

qa-failure

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Command failed (workunit test suites/fsstress.sh) in [1]. Couldn't get more details as the editor hangs because of the huge file size of teuthology.log.

[1] https://pulpito.ceph.com/yuriw-2023-02-16_19:08:52-fs-wip-yuri3-testing-2023-02-16-0752-quincy-distro-default-smithi/7176561/

2023-02-17T01:54:41.455 INFO:teuthology.orchestra.run.smithi026.stderr:2023-02-17T01:54:41.452+0000 7fbeda7fc700  1 -- 172.21.15.26:0/3418241292 <== osd.0 v2:172.21.15.26\
:6816/1365961 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbeec15b2f0 con 0x7fbeec1506f0
2023-02-17T01:54:41.456 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/run_tasks.py", line 103, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/run_tasks.py", line 82, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 145, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env,
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 424, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi026 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.\
0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fd249f18f047d5b86a29796a05bfeba7a7666d84 TESTDIR="/home/ubuntu/cephtest" C\
EPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ub\
untu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2023-02-17T01:54:42.751 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b55ee7e8797844639058152a9e45c65a
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/run_tasks.py", line 103, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/run_tasks.py", line 82, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 145, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env,
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/workunit.py", line 424, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi026 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.\
0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fd249f18f047d5b86a29796a05bfeba7a7666d84 TESTDIR="/home/ubuntu/cephtest" C\
EPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ub\
untu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2023-02-17T01:54:42.754 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds