Project

General

Profile

Actions

Bug #45721

closed

CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test

Added by Brad Hubbard almost 4 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

100%

Source:
Q/A
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2020-05-24_19:30:40-rados-wip-yuri-master_5.24.20-distro-basic-smithi/5088170

2020-05-24T22:42:37.494 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.TestWatchNotify.test ... FAIL
2020-05-24T22:42:37.495 INFO:tasks.workunit.client.0.smithi082.stderr:Traceback (most recent call last):
2020-05-24T22:42:37.495 INFO:tasks.workunit.client.0.smithi082.stderr:  File "rados.pyx", line 800, in rados.Rados.require_state
2020-05-24T22:42:37.495 INFO:tasks.workunit.client.0.smithi082.stderr:rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
2020-05-24T22:42:37.495 INFO:tasks.workunit.client.0.smithi082.stderr:Exception ignored in: 'rados.Watch.__dealloc__'
2020-05-24T22:42:37.496 INFO:tasks.workunit.client.0.smithi082.stderr:Traceback (most recent call last):
2020-05-24T22:42:37.496 INFO:tasks.workunit.client.0.smithi082.stderr:  File "rados.pyx", line 800, in rados.Rados.require_state
2020-05-24T22:42:37.496 INFO:tasks.workunit.client.0.smithi082.stderr:rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
2020-05-24T22:42:37.496 INFO:tasks.workunit.client.0.smithi082.stderr:Traceback (most recent call last):
2020-05-24T22:42:37.496 INFO:tasks.workunit.client.0.smithi082.stderr:  File "rados.pyx", line 800, in rados.Rados.require_state
2020-05-24T22:42:37.497 INFO:tasks.workunit.client.0.smithi082.stderr:rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
2020-05-24T22:42:37.497 INFO:tasks.workunit.client.0.smithi082.stderr:Exception ignored in: 'rados.Watch.__dealloc__'
2020-05-24T22:42:37.497 INFO:tasks.workunit.client.0.smithi082.stderr:Traceback (most recent call last):
2020-05-24T22:42:37.497 INFO:tasks.workunit.client.0.smithi082.stderr:  File "rados.pyx", line 800, in rados.Rados.require_state
2020-05-24T22:42:37.497 INFO:tasks.workunit.client.0.smithi082.stderr:rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
2020-05-24T22:42:37.504 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.test_rados_init_error ... ok
2020-05-24T22:42:37.583 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.test_rados_init ... ok
2020-05-24T22:42:37.607 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.test_ioctx_context_manager ... ok
2020-05-24T22:42:37.618 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.test_parse_argv ... ok
2020-05-24T22:42:37.623 INFO:tasks.workunit.client.0.smithi082.stderr:test_rados.test_parse_argv_empty_str ... ok
2020-05-24T22:42:37.624 INFO:tasks.workunit.client.0.smithi082.stderr:
2020-05-24T22:42:37.624 INFO:tasks.workunit.client.0.smithi082.stderr:======================================================================
2020-05-24T22:42:37.624 INFO:tasks.workunit.client.0.smithi082.stderr:FAIL: test_rados.TestWatchNotify.test
2020-05-24T22:42:37.624 INFO:tasks.workunit.client.0.smithi082.stderr:----------------------------------------------------------------------
2020-05-24T22:42:37.625 INFO:tasks.workunit.client.0.smithi082.stderr:Traceback (most recent call last):
2020-05-24T22:42:37.625 INFO:tasks.workunit.client.0.smithi082.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2020-05-24T22:42:37.625 INFO:tasks.workunit.client.0.smithi082.stderr:    self.test(*self.arg)
2020-05-24T22:42:37.625 INFO:tasks.workunit.client.0.smithi082.stderr:  File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1290, in test
2020-05-24T22:42:37.625 INFO:tasks.workunit.client.0.smithi082.stderr:    eq(self.notify_cnt[watch_id1], 3)
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:AssertionError: 4 != 3
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:----------------------------------------------------------------------
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:Ran 81 tests in 293.733s
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:
2020-05-24T22:42:37.626 INFO:tasks.workunit.client.0.smithi082.stderr:FAILED (failures=1)
2020-05-24T22:42:37.645 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-05-24T22:42:37.646 INFO:tasks.workunit:Stopping ['rados/test_python.sh'] on client.0...
2020-05-24T22:42:37.646 INFO:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2020-05-24T22:42:37.893 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/run_tasks.py", line 87, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/run_tasks.py", line 66, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.24.20/qa/tasks/workunit.py", line 127, in task
    coverage_and_limits=not config.get('no_coverage_and_limits', None))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/parallel.py", line 87, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/parallel.py", line 101, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/parallel.py", line 37, in resurrect_traceback
    reraise(*exc_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/parallel.py", line 24, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-master_5.24.20/qa/tasks/workunit.py", line 415, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/remote.py", line 206, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 473, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test rados/test_python.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CE
PH_REF=9dce54cc5473ddca3d64786e557f4b0c097deed7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adj
ust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh'

Related issues 4 (1 open3 closed)

Related to RADOS - Bug #52553: pybind: rados.RadosStateError raised when closed watch object goes out of scope after cluster shutdownResolvedTim Serong

Actions
Related to RADOS - Bug #53789: CommandFailedError (rados/test_python.sh): "RADOS object not found" causes test_rados.TestWatchNotify.test_aio_notify to failPending BackportRadoslaw Zarzynski

Actions
Copied to RADOS - Backport #57544: pacific: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.testResolvedNitzan MordechaiActions
Copied to RADOS - Backport #57545: quincy: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.testResolvedNitzan MordechaiActions
Actions

Also available in: Atom PDF