https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2020-06-05T16:20:03Z
Ceph
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=167606
2020-06-05T16:20:03Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2020-06-04_18:03:48-rados-wip-yuri2-testing-2020-06-03-2341-MASTER-distro-basic-smithi/5117777</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=168877
2020-06-24T02:47:26Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><pre>
2020-06-23T23:35:22.984 INFO:tasks.workunit.client.0.smithi107.stderr:======================================================================
2020-06-23T23:35:22.984 INFO:tasks.workunit.client.0.smithi107.stderr:FAIL: test_rados.TestWatchNotify.test
2020-06-23T23:35:22.984 INFO:tasks.workunit.client.0.smithi107.stderr:----------------------------------------------------------------------
2020-06-23T23:35:22.985 INFO:tasks.workunit.client.0.smithi107.stderr:Traceback (most recent call last):
2020-06-23T23:35:22.985 INFO:tasks.workunit.client.0.smithi107.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2020-06-23T23:35:22.985 INFO:tasks.workunit.client.0.smithi107.stderr: self.test(*self.arg)
2020-06-23T23:35:22.985 INFO:tasks.workunit.client.0.smithi107.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1281, in test
2020-06-23T23:35:22.985 INFO:tasks.workunit.client.0.smithi107.stderr: eq(self.notify_cnt[watch_id1], 2)
2020-06-23T23:35:22.986 INFO:tasks.workunit.client.0.smithi107.stderr:AssertionError: 3 != 2
</pre>
<p>/a/kchai-2020-06-23_15:15:59-rados-wip-kefu-testing-2020-06-23-1353-distro-basic-smithi/5173648/</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=169184
2020-06-27T08:25:24Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><pre>
2020-06-27T08:17:29.361 INFO:tasks.workunit.client.0.smithi203.stderr:FAIL: test_rados.TestWatchNotify.test
2020-06-27T08:17:29.361 INFO:tasks.workunit.client.0.smithi203.stderr:----------------------------------------------------------------------
2020-06-27T08:17:29.362 INFO:tasks.workunit.client.0.smithi203.stderr:Traceback (most recent call last):
2020-06-27T08:17:29.362 INFO:tasks.workunit.client.0.smithi203.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2020-06-27T08:17:29.362 INFO:tasks.workunit.client.0.smithi203.stderr: self.test(*self.arg)
2020-06-27T08:17:29.363 INFO:tasks.workunit.client.0.smithi203.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1281, in test
2020-06-27T08:17:29.363 INFO:tasks.workunit.client.0.smithi203.stderr: eq(self.notify_cnt[watch_id1], 2)
2020-06-27T08:17:29.363 INFO:tasks.workunit.client.0.smithi203.stderr:AssertionError: 3 != 2
</pre>
<p>/a//kchai-2020-06-27_07:37:00-rados-wip-kefu-testing-2020-06-27-1407-distro-basic-smithi/5183650</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=172301
2020-08-04T18:29:11Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} tasks/rados_python}</p>
<p>/a/nojha-2020-07-31_22:50:07-rados-wip-35628-2020-07-28-distro-basic-smithi/5273950</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=181492
2020-12-17T06:45:34Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><pre>
2020-12-16T21:11:22.044 INFO:tasks.workunit.client.0.smithi167.stderr:FAIL: test_rados.TestWatchNotify.test
2020-12-16T21:11:22.045 INFO:tasks.workunit.client.0.smithi167.stderr:----------------------------------------------------------------------
2020-12-16T21:11:22.045 INFO:tasks.workunit.client.0.smithi167.stderr:Traceback (most recent call last):
2020-12-16T21:11:22.045 INFO:tasks.workunit.client.0.smithi167.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2020-12-16T21:11:22.045 INFO:tasks.workunit.client.0.smithi167.stderr: self.test(*self.arg)
2020-12-16T21:11:22.045 INFO:tasks.workunit.client.0.smithi167.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1348, in test
2020-12-16T21:11:22.046 INFO:tasks.workunit.client.0.smithi167.stderr: eq(self.notify_cnt[watch_id1], 1)
2020-12-16T21:11:22.046 INFO:tasks.workunit.client.0.smithi167.stderr:AssertionError: 2 != 1
</pre>
<p>/a/kchai-2020-12-16_15:58:13-rados-wip-kefu-testing-2020-12-16-1740-distro-basic-smithi/5713555</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=184244
2021-02-03T19:11:18Z
Neha Ojha
nojha@redhat.com
<ul></ul><pre>
2021-02-02T02:15:24.737 INFO:tasks.workunit.client.0.smithi036.stderr:======================================================================
2021-02-02T02:15:24.737 INFO:tasks.workunit.client.0.smithi036.stderr:FAIL: test_rados.TestWatchNotify.test
2021-02-02T02:15:24.737 INFO:tasks.workunit.client.0.smithi036.stderr:----------------------------------------------------------------------
2021-02-02T02:15:24.738 INFO:tasks.workunit.client.0.smithi036.stderr:Traceback (most recent call last):
2021-02-02T02:15:24.738 INFO:tasks.workunit.client.0.smithi036.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
2021-02-02T02:15:24.738 INFO:tasks.workunit.client.0.smithi036.stderr: self.test(*self.arg)
2021-02-02T02:15:24.738 INFO:tasks.workunit.client.0.smithi036.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1406, in test
2021-02-02T02:15:24.738 INFO:tasks.workunit.client.0.smithi036.stderr: eq(self.notify_cnt[watch_id1], 3)
2021-02-02T02:15:24.739 INFO:tasks.workunit.client.0.smithi036.stderr:AssertionError: 5 != 3
2021-02-02T02:15:24.739 INFO:tasks.workunit.client.0.smithi036.stderr:
2021-02-02T02:15:24.739 INFO:tasks.workunit.client.0.smithi036.stderr:----------------------------------------------------------------------
2021-02-02T02:15:24.739 INFO:tasks.workunit.client.0.smithi036.stderr:Ran 88 tests in 313.846s
2021-02-02T02:15:24.739 INFO:tasks.workunit.client.0.smithi036.stderr:
2021-02-02T02:15:24.740 INFO:tasks.workunit.client.0.smithi036.stderr:FAILED (failures=1)
</pre>
<p>/a/nojha-2021-02-01_21:31:14-rados-wip-39145-distro-basic-smithi/5846891</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=187518
2021-03-12T18:20:31Z
Deepika Upadhyay
<ul></ul><pre>
2021-03-11T17:24:39.745 INFO:tasks.workunit.client.0.smithi123.stderr:======================================================================
2021-03-11T17:24:39.746 INFO:tasks.workunit.client.0.smithi123.stderr:FAIL: test_rados.TestWatchNotify.test
2021-03-11T17:24:39.746 INFO:tasks.workunit.client.0.smithi123.stderr:----------------------------------------------------------------------
2021-03-11T17:24:39.746 INFO:tasks.workunit.client.0.smithi123.stderr:Traceback (most recent call last):
2021-03-11T17:24:39.746 INFO:tasks.workunit.client.0.smithi123.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2021-03-11T17:24:39.747 INFO:tasks.workunit.client.0.smithi123.stderr: self.test(*self.arg)
2021-03-11T17:24:39.747 INFO:tasks.workunit.client.0.smithi123.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1276, in test
2021-03-11T17:24:39.747 INFO:tasks.workunit.client.0.smithi123.stderr: eq(self.notify_cnt[watch_id1], 2)
2021-03-11T17:24:39.747 INFO:tasks.workunit.client.0.smithi123.stderr:AssertionError: 4 != 2
2021-03-11T17:24:39.747 INFO:tasks.workunit.client.0.smithi123.stderr:
2021-03-11T17:24:39.748 INFO:tasks.workunit.client.0.smithi123.stderr:----------------------------------------------------------------------
2021-03-11T17:24:39.748 INFO:tasks.workunit.client.0.smithi123.stderr:Ran 80 tests in 289.328s
</pre>
<p>/ceph/teuthology-archive/yuriw-2021-03-09_20:27:38-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-smithi/5950464/teuthology.log<br />/ceph/teuthology-archive/yuriw-2021-03-09_20:28:34-rados-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951087/teuthology.log</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=189394
2021-04-02T20:20:45Z
Sage Weil
sage@newdream.net
<ul></ul><p>/a/sage-2021-04-02_14:05:32-rados-wip-sage2-testing-2021-04-01-1417-pacific-distro-basic-smithi/6017345</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=189395
2021-04-02T20:21:04Z
Sage Weil
sage@newdream.net
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=189476
2021-04-05T13:40:15Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><pre>
2021-04-05T12:33:34.093 INFO:tasks.workunit.client.0.gibba010.stderr:======================================================================
2021-04-05T12:33:34.094 INFO:tasks.workunit.client.0.gibba010.stderr:FAIL: test_rados.TestWatchNotify.test
2021-04-05T12:33:34.094 INFO:tasks.workunit.client.0.gibba010.stderr:----------------------------------------------------------------------
2021-04-05T12:33:34.094 INFO:tasks.workunit.client.0.gibba010.stderr:Traceback (most recent call last):
2021-04-05T12:33:34.094 INFO:tasks.workunit.client.0.gibba010.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2021-04-05T12:33:34.094 INFO:tasks.workunit.client.0.gibba010.stderr: self.test(*self.arg)
2021-04-05T12:33:34.095 INFO:tasks.workunit.client.0.gibba010.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1397, in test
2021-04-05T12:33:34.095 INFO:tasks.workunit.client.0.gibba010.stderr: eq(self.notify_cnt[watch_id1], 2)
2021-04-05T12:33:34.095 INFO:tasks.workunit.client.0.gibba010.stderr:AssertionError: 3 != 2
2021-04-05T12:33:34.095 INFO:tasks.workunit.client.0.gibba010.stderr:
2021-04-05T12:33:34.096 INFO:tasks.workunit.client.0.gibba010.stderr:----------------------------------------------------------------------
</pre>
<p>/a/kchai-2021-04-05_10:46:36-rados-wip-kefu-testing-2021-04-05-1650-distro-basic-gibba/6021723/</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=190291
2021-04-13T19:30:22Z
Greg Farnum
gfarnum@redhat.com
<ul></ul><p><a class="external" href="https://pulpito.ceph.com/gregf-2021-04-13_09:22:15-rados-wip-stretch-mon-state-412-distro-basic-smithi/6043691">https://pulpito.ceph.com/gregf-2021-04-13_09:22:15-rados-wip-stretch-mon-state-412-distro-basic-smithi/6043691</a></p>
<pre>======================================================================
FAIL: test_rados.TestWatchNotify.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1406, in test
eq(self.notify_cnt[watch_id1], 3)
AssertionError: 4 != 3
</pre>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=193573
2021-04-30T16:27:21Z
Neha Ojha
nojha@redhat.com
<ul><li><strong>Backport</strong> set to <i>pacific</i></li></ul><p>/a/yuriw-2021-04-30_12:58:14-rados-wip-yuri2-testing-2021-04-29-1501-pacific-distro-basic-smithi/6086154</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=195145
2021-05-14T14:02:59Z
Deepika Upadhyay
<ul><li><strong>Backport</strong> changed from <i>pacific</i> to <i>octopus, pacific</i></li></ul><pre>
2021-05-14T00:02:12.634 INFO:tasks.workunit.client.0.smithi028.stderr:Exception ignored in: 'rados.Watch.__dealloc__'
2021-05-14T00:02:12.634 INFO:tasks.workunit.client.0.smithi028.stderr:Traceback (most recent call last):
2021-05-14T00:02:12.635 INFO:tasks.workunit.client.0.smithi028.stderr: File "rados.pyx", line 799, in rados.Rados.require_state
2021-05-14T00:02:12.635 INFO:tasks.workunit.client.0.smithi028.stderr:rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state s
</pre><br /><pre>
2021-05-14T00:02:12.729 INFO:tasks.workunit.client.0.smithi028.stderr:======================================================================
2021-05-14T00:02:12.729 INFO:tasks.workunit.client.0.smithi028.stderr:FAIL: test_rados.TestWatchNotify.test
2021-05-14T00:02:12.729 INFO:tasks.workunit.client.0.smithi028.stderr:----------------------------------------------------------------------
2021-05-14T00:02:12.729 INFO:tasks.workunit.client.0.smithi028.stderr:Traceback (most recent call last):
2021-05-14T00:02:12.730 INFO:tasks.workunit.client.0.smithi028.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
2021-05-14T00:02:12.730 INFO:tasks.workunit.client.0.smithi028.stderr: self.test(*self.arg)
2021-05-14T00:02:12.730 INFO:tasks.workunit.client.0.smithi028.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1276, in test
2021-05-14T00:02:12.730 INFO:tasks.workunit.client.0.smithi028.stderr: eq(self.notify_cnt[watch_id1], 2)
2021-05-14T00:02:12.731 INFO:tasks.workunit.client.0.smithi028.stderr:AssertionError: 3 != 2
</pre>
<p>/ceph/teuthology-archive/yuriw-2021-05-13_23:05:56-rados-wip-yuri4-testing-2021-05-13-1301-octopus-distro-basic-smithi/6114277/teuthology.log</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=196372
2021-06-02T18:46:57Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2021-05-25_19:21:19-rados-wip-yuri2-testing-2021-05-25-0940-pacific-distro-basic-smithi/6134490</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=198118
2021-06-29T16:59:23Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2021-06-28_17:32:48-rados-wip-yuri2-testing-2021-06-28-0858-pacific-distro-basic-smithi/6239575</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=198132
2021-06-30T06:55:01Z
Deepika Upadhyay
<ul></ul><p><a class="external" href="http://qa-proxy.ceph.com/teuthology/ideepika-2021-06-30_04:28:06-rados-wip-yuri7-testing-2021-06-28-1224-octopus-distro-basic-smithi/6245147/teuthology.log">http://qa-proxy.ceph.com/teuthology/ideepika-2021-06-30_04:28:06-rados-wip-yuri7-testing-2021-06-28-1224-octopus-distro-basic-smithi/6245147/teuthology.log</a></p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=198512
2021-07-06T17:38:02Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2021-07-02_17:35:44-rados-pacific-distro-basic-smithi/6249971</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=199384
2021-07-20T15:33:33Z
Deepika Upadhyay
<ul></ul><p><a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021-07-16-0906-octopus-distro-basic-smithi/6276528/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021-07-16-0906-octopus-distro-basic-smithi/6276528/teuthology.log</a></p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=200355
2021-08-03T00:23:13Z
Brad Hubbard
bhubbard@redhat.com
<ul></ul><p>I've modified this test to only run the TestWatchNotify subtests (2) and to generate debug logging. I'll report back what, if anything, I find out.</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=200797
2021-08-09T17:49:06Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2021-08-06_16:31:19-rados-wip-yuri-master-8.6.21-distro-basic-smithi/6324539</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=201787
2021-08-24T18:31:25Z
Neha Ojha
nojha@redhat.com
<ul></ul><p>/a/yuriw-2021-08-23_19:24:05-rados-wip-yuri4-testing-2021-08-23-0812-pacific-distro-basic-smithi/6353883</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=203372
2021-09-21T11:43:00Z
Deepika Upadhyay
<ul></ul><pre>
2021-09-17T07:03:13.050 INFO:tasks.workunit.client.0.smithi026.stderr:FAIL: test_rados.TestWatchNotify.test
2021-09-17T07:03:13.051 INFO:tasks.workunit.client.0.smithi026.stderr:----------------------------------------------------------------------
2021-09-17T07:03:13.051 INFO:tasks.workunit.client.0.smithi026.stderr:Traceback (most recent call last):
2021-09-17T07:03:13.051 INFO:tasks.workunit.client.0.smithi026.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2021-09-17T07:03:13.051 INFO:tasks.workunit.client.0.smithi026.stderr: self.test(*self.arg)
2021-09-17T07:03:13.052 INFO:tasks.workunit.client.0.smithi026.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1285, in test
2021-09-17T07:03:13.052 INFO:tasks.workunit.client.0.smithi026.stderr: eq(self.notify_cnt[watch_id1], 3)
2021-09-17T07:03:13.052 INFO:tasks.workunit.client.0.smithi026.stderr:AssertionError: 4 != 3
</pre>
<p>/ceph/teuthology-archive/yuriw-2021-09-16_21:30:19-rados-wip-yuri5-testing-2021-09-16-1056-octopus-distro-basic-smithi/6394002/teuthology.log</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=203382
2021-09-21T13:47:09Z
Deepika Upadhyay
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-5 priority-high3 closed" href="/issues/52553">Bug #52553</a>: pybind: rados.RadosStateError raised when closed watch object goes out of scope after cluster shutdown</i> added</li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=208049
2022-01-06T22:03:28Z
Laura Flores
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-14 priority-5 priority-high3" href="/issues/53789">Bug #53789</a>: CommandFailedError (rados/test_python.sh): "RADOS object not found" causes test_rados.TestWatchNotify.test_aio_notify to fail</i> added</li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=208194
2022-01-11T05:29:25Z
Sridhar Seshasayee
<ul></ul><p>/a/yuriw-2022-01-06_15:57:04-rados-wip-yuri6-testing-2022-01-05-1255-distro-default-smithi/6599449</p>
<pre>
2022-01-07T02:11:42.882 INFO:tasks.workunit.client.0.smithi067.stderr:======================================================================
2022-01-07T02:11:42.883 INFO:tasks.workunit.client.0.smithi067.stderr:FAIL: test_rados.TestWatchNotify.test
2022-01-07T02:11:42.883 INFO:tasks.workunit.client.0.smithi067.stderr:----------------------------------------------------------------------
2022-01-07T02:11:42.883 INFO:tasks.workunit.client.0.smithi067.stderr:Traceback (most recent call last):
2022-01-07T02:11:42.883 INFO:tasks.workunit.client.0.smithi067.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
2022-01-07T02:11:42.884 INFO:tasks.workunit.client.0.smithi067.stderr: self.test(*self.arg)
2022-01-07T02:11:42.884 INFO:tasks.workunit.client.0.smithi067.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1451, in test
2022-01-07T02:11:42.884 INFO:tasks.workunit.client.0.smithi067.stderr: eq(self.notify_cnt[watch_id1], 3)
2022-01-07T02:11:42.884 INFO:tasks.workunit.client.0.smithi067.stderr:AssertionError: 5 != 3
</pre>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=208625
2022-01-18T16:23:44Z
Laura Flores
<ul></ul><p>/a/yuriw-2022-01-14_23:22:09-rados-wip-yuri6-testing-2022-01-14-1207-distro-default-smithi/6617813</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=209198
2022-01-25T17:47:54Z
Neha Ojha
nojha@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Laura Flores</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=211033
2022-02-23T17:00:43Z
Laura Flores
<ul></ul><p>/a/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698327</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=211458
2022-03-03T18:15:15Z
Laura Flores
<ul></ul><p>/a/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714724</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=211677
2022-03-09T19:19:47Z
Radoslaw Zarzynski
rzarzyns@redhat.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=211708
2022-03-10T04:59:29Z
Kamoltat (Junior) Sirivadhna
<ul></ul><p>/a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721329</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=215421
2022-04-28T18:59:56Z
Laura Flores
<ul></ul><p>/a/yuriw-2022-04-27_02:52:22-rados-pacific-distro-default-smithi/6807766</p>
<pre><code class="text syntaxhl"><span class="CodeRay">2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr:======================================================================
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr:FAIL: test_rados.TestWatchNotify.test
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr:----------------------------------------------------------------------
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr:Traceback (most recent call last):
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr: File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr: self.test(*self.arg)
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1397, in test
2022-04-27T03:49:25.747 INFO:tasks.workunit.client.0.smithi045.stderr: eq(self.notify_cnt[watch_id1], 2)
2022-04-27T03:49:25.748 INFO:tasks.workunit.client.0.smithi045.stderr:AssertionError: 3 != 2
2022-04-27T03:49:25.748 INFO:tasks.workunit.client.0.smithi045.stderr:
2022-04-27T03:49:25.748 INFO:tasks.workunit.client.0.smithi045.stderr:----------------------------------------------------------------------
</span></code></pre>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=215994
2022-05-11T05:20:09Z
Nitzan Mordechai
nmordech@redhat.com
<ul></ul><p>/home/teuthworker/archive/yuriw-2022-04-29_15:44:49-rados-wip-yuri5-testing-2022-04-28-1007-distro-default-smithi/6813940</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=216194
2022-05-13T22:49:25Z
Laura Flores
<ul></ul><p>I reproduced the symptoms of this bug locally by incrementing the notify count before an eq check. The extra increment causes the "shutdown" rados state and the Assertion failure. This indicates that the notify_cnt is getting incremented at some point when it shouldn't.</p>
<p>diff in the code (for testing)<br /><pre><code class="diff syntaxhl"><span class="CodeRay"> assert(self.ioctx.notify(self.OID, 'best'))
with self.lock:
<span class="line insert"><span class="insert">+</span> self.notify_cnt[watch_id1] += 1 # for debugging</span>
eq(self.notify_cnt[watch_id1], 2)
eq(self.notify_cnt[watch_id2], 2)
eq(self.notify_data[watch_id1], b'best')
eq(self.notify_data[watch_id2], b'best')
</span></code></pre></p>
<p>../qa/workunits/rados/test_python.sh<br /><pre><code class="text syntaxhl"><span class="CodeRay">test_rados.TestWatchNotify.test ... FAIL
Traceback (most recent call last):
File "rados.pyx", line 482, in rados.Rados.require_state
rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
Exception ignored in: 'rados.Watch.__dealloc__'
Traceback (most recent call last):
File "rados.pyx", line 482, in rados.Rados.require_state
rados.RadosStateError: RADOS rados state (You cannot perform that operation on a Rados object in state shutdown.)
test_rados.TestWatchNotify.test_aio_notify ... ok
test_rados.test_rados_init_error ... ok
test_rados.test_rados_init ... ok
test_rados.test_ioctx_context_manager ... ok
test_rados.test_parse_argv ... ok
test_rados.test_parse_argv_empty_str ... ok
======================================================================
FAIL: test_rados.TestWatchNotify.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/lflores/ceph/src/test/pybind/test_rados.py", line 1443, in test
eq(self.notify_cnt[watch_id1], 2)
AssertionError: 3 != 2
</span></code></pre></p>
<p>A previous patch (tracked by <a class="issue tracker-1 status-3 priority-5 priority-high3 closed" title="Bug: pybind: rados.RadosStateError raised when closed watch object goes out of scope after cluster shu... (Resolved)" href="https://tracker.ceph.com/issues/52553">#52553</a>) tried fixing this symptom by modifying <i>dealloc</i>, but the problem persists. So, a fix might involve decrementing notify_cnt whenever a false "notify" is counted.</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=216701
2022-05-25T18:27:40Z
Radoslaw Zarzynski
rzarzyns@redhat.com
<ul></ul><p>Let me paste a Laura's comment from <a class="external" href="https://github.com/ceph/ceph/pull/45825">https://github.com/ceph/ceph/pull/45825</a>:</p>
<blockquote>
<p>@NitzanMordhai perhaps similar logic can be applied for this Tracker: <a class="external" href="https://tracker.ceph.com/issues/45721">https://tracker.ceph.com/issues/45721</a>. Let me know if you have any thoughts!</p>
</blockquote>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=216794
2022-05-27T15:35:52Z
Laura Flores
<ul></ul><p>/a/yuriw-2022-05-26_23:23:48-rados-wip-yuri2-testing-2022-05-26-1430-quincy-distro-default-smithi/6849426$</p>
<pre><code class="text syntaxhl"><span class="CodeRay">2022-05-26T23:55:09.585 INFO:tasks.workunit.client.0.smithi105.stderr:======================================================================
2022-05-26T23:55:09.585 INFO:tasks.workunit.client.0.smithi105.stderr:FAIL: test_rados.TestWatchNotify.test
2022-05-26T23:55:09.585 INFO:tasks.workunit.client.0.smithi105.stderr:----------------------------------------------------------------------
2022-05-26T23:55:09.585 INFO:tasks.workunit.client.0.smithi105.stderr:Traceback (most recent call last):
2022-05-26T23:55:09.585 INFO:tasks.workunit.client.0.smithi105.stderr: File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr: self.test(*self.arg)
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr: File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_rados.py", line 1451, in test
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr: eq(self.notify_cnt[watch_id1], 3)
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr:AssertionError: 4 != 3
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr:
2022-05-26T23:55:09.586 INFO:tasks.workunit.client.0.smithi105.stderr:----------------------------------------------------------------------
</span></code></pre>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=217491
2022-06-08T07:53:51Z
Nitzan Mordechai
nmordech@redhat.com
<ul></ul><p>That could work, but when we have socket failure injection, the error callback will not be calling in the python API (unlike librados)<br />I think we need to check the API calls itself (and your suggestion for the test fix)</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=217777
2022-06-10T16:48:02Z
Laura Flores
<ul></ul><p>Nitzan Mordechai wrote:</p>
<blockquote>
<p>That could work, but when we have socket failure injection, the error callback will not be calling in the python API (unlike librados)<br />I think we need to check the API calls itself (and your suggestion for the test fix)</p>
</blockquote>
<p>Thanks for the insight. Still happening occasionally in teuthology runs, so I will try out some potential fixes.</p>
<p>/a/yuriw-2022-06-10_03:10:47-rados-wip-yuri4-testing-2022-06-09-1510-quincy-distro-default-smithi/6871849</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=222998
2022-08-09T08:08:16Z
Sridhar Seshasayee
<ul></ul><p>Seeing this in a Quincy run:<br />/a/yuriw-2022-08-08_22:19:32-rados-wip-yuri4-testing-2022-08-08-1009-quincy-distro-default-smithi/6961889</p>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=223536
2022-08-16T08:02:36Z
Radoslaw Zarzynski
rzarzyns@redhat.com
<ul><li><strong>Assignee</strong> changed from <i>Laura Flores</i> to <i>Nitzan Mordechai</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=223724
2022-08-18T11:44:26Z
Nitzan Mordechai
nmordech@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Pull request ID</strong> set to <i>47668</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=224245
2022-08-24T18:36:46Z
Radoslaw Zarzynski
rzarzyns@redhat.com
<ul><li><strong>Backport</strong> changed from <i>octopus, pacific</i> to <i>pacific,quincy</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=224359
2022-08-26T15:53:37Z
Laura Flores
<ul><li><strong>Tags</strong> set to <i>test-failure</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=225621
2022-09-14T19:16:59Z
Neha Ojha
nojha@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=225623
2022-09-14T19:24:13Z
Backport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/57544">Backport #57544</a>: pacific: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test</i> added</li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=225625
2022-09-14T19:24:20Z
Backport Bot
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/57545">Backport #57545</a>: quincy: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test</i> added</li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=225627
2022-09-14T19:24:22Z
Backport Bot
<ul><li><strong>Tags</strong> set to <i>backport_processed</i></li></ul>
RADOS - Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_rados.TestWatchNotify.test
https://tracker.ceph.com/issues/45721?journal_id=250452
2023-11-18T19:52:58Z
Konstantin Shalygin
k0ste@k0ste.ru
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul>