Ceph » Orchestrator

teuthology.exceptions.CommandFailedError: Command failed with status 1: ['../src/stop.sh']

This API test failure has been occurring across PR Jenkins builds. It appears to show up sporadically-- occasionally a PR will pass the test, but others will fail. This failure does not seem related to any changes in the PRs in which it occurs.

Here, I have copied the Python Traceback as well as some surrounding output to provide more context.

Collecting pytz
  Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Installing collected packages: more-itertools, pytz, jaraco.functools, tempora, repoze.lru, portend, idna, cheroot, chardet, Routes, requests, pyopenssl, PyJWT, CherryPy, ceph, bcrypt
  Attempting uninstall: idna
    Found existing installation: idna 3.3
    Uninstalling idna-3.3:
      Successfully uninstalled idna-3.3
  Attempting uninstall: requests
    Found existing installation: requests 2.26.0
    Uninstalling requests-2.26.0:
      Successfully uninstalled requests-2.26.0
  Attempting uninstall: PyJWT
    Found existing installation: PyJWT 2.3.0
    Uninstalling PyJWT-2.3.0:
      Successfully uninstalled PyJWT-2.3.0
  Running setup.py develop for ceph
  Attempting uninstall: bcrypt
    Found existing installation: bcrypt 3.2.0
    Uninstalling bcrypt-3.2.0:
      Successfully uninstalled bcrypt-3.2.0
Successfully installed CherryPy-13.1.0 PyJWT-2.0.1 Routes-2.4.1 bcrypt-3.1.4 ceph-1.0.0 chardet-4.0.0 cheroot-8.5.2 idna-2.10 jaraco.functools-3.3.0 more-itertools-4.1.0 portend-3.0.0 pyopenssl-21.0.0 pytz-2021.3 repoze.lru-0.7 requests-2.25.1 tempora-4.1.2
/tmp/tmp.mAQFsq4fJ8
Processing /home/jenkins-build/.cache/pip/wheels/d8/81/0a/fae9efd3c9c706cefa25842310896e727a46567f2dc2dac6a8/coverage-4.5.2-cp38-cp38-linux_x86_64.whl
Installing collected packages: coverage
Successfully installed coverage-4.5.2
Cannot find device "ceph-brx" 
2021-10-21 14:21:10,353.353 INFO:__main__:Creating cluster with 1 MDS daemons
2021-10-21 14:21:10,354.354 INFO:__main__:
tearing down the cluster...
rm: cannot remove '/var/run/ceph': Permission denied
Using guessed paths /home/jenkins-build/build/workspace/ceph-api/build/lib/ ['/home/jenkins-build/build/workspace/ceph-api/qa', '/home/jenkins-build/build/workspace/ceph-api/build/lib/cython_modules/lib.3', '/home/jenkins-build/build/workspace/ceph-api/src/pybind']
Traceback (most recent call last):
  File "../qa/tasks/vstart_runner.py", line 1522, in <module>
    exec_test()
  File "../qa/tasks/vstart_runner.py", line 1357, in exec_test
    teardown_cluster()
  File "../qa/tasks/vstart_runner.py", line 1091, in teardown_cluster
    remote.run(args=[os.path.join(SRC_PREFIX, "stop.sh")], timeout=60)
  File "../qa/tasks/vstart_runner.py", line 410, in run
    return self._do_run(**kwargs)
  File "../qa/tasks/vstart_runner.py", line 478, in _do_run
    proc.wait()
  File "../qa/tasks/vstart_runner.py", line 221, in wait
    raise CommandFailedError(self.args, self.exitstatus)
teuthology.exceptions.CommandFailedError: Command failed with status 1: ['../src/stop.sh']
find: ‘/home/jenkins-build/build/workspace/ceph-api/build/out’: No such file or directory

• https://jenkins.ceph.com/job/ceph-api/26288/consoleFull#962382583c212b007-e891-4176-9ee7-2f60eca393b7