Bug #44233
Updated by Laura Paduano about 4 years ago
<pre><code class="text">
2020-02-21T06:57:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.361+0000 7ff6067fc700 1 -- 172.21.15.71:0/378048796 <== mgr.11302 v2:172.21.15.71:6808/62702 1 ==== mgr_command_reply(tid 0: -22 TypeError: 'partialmethod' object is not callable) v1 ==== 57+0+0 (crc 0 0 0) 0x7ff61805f0b0 con 0x7ff5f80560c0
2020-02-21T06:57:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.361+0000 7ff6067fc700 20 mgrc handle_command_reply tid 0 r -22
2020-02-21T06:57:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 10 mgrc shutdown
2020-02-21T06:57:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:6808/62702,v1:172.21.15.71:6809/62702] conn(0x7ff5f80560c0 msgr2=0x7ff5f8058580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-02-21T06:57:29.370 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 --2- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:6808/62702,v1:172.21.15.71:6809/62702] conn(0x7ff5f80560c0 0x7ff5f8058580 crc :-1 s=READY pgs=21 cs=0 l=1 rx=0 tx=0).stop
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618097740 msgr2=0x7ff61813f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 --2- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618097740 0x7ff61813f540 secure :-1 s=READY pgs=2206 cs=0 l=1 rx=0x7ff608002f30 tx=0x7ff618060de0).stop
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 shutdown_connections
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 --2- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:6808/62702,v1:172.21.15.71:6809/62702] conn(0x7ff5f80560c0 0x7ff5f8058580 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 --2- 172.21.15.71:0/378048796 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff618097740 0x7ff61813f540 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rx=0 tx=0).stop
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 --2- 172.21.15.71:0/378048796 >> v2:172.21.15.42:3300/0 conn(0x7ff6180a53e0 0x7ff6180634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 >> 172.21.15.71:0/378048796 conn(0x7ff61805bea0 msgr2=0x7ff6180a6710 unknown :-1 s=STATE_NONE l=0).mark_down
2020-02-21T06:57:29.371 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 shutdown_connections
2020-02-21T06:57:29.372 INFO:teuthology.orchestra.run.smithi071.stderr:2020-02-21T06:57:29.362+0000 7ff61f7ae700 1 -- 172.21.15.71:0/378048796 wait complete.
2020-02-21T06:57:29.372 INFO:teuthology.orchestra.run.smithi071.stderr:Error EINVAL: TypeError: 'partialmethod' object is not callable
2020-02-21T06:57:29.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2020-02-21T06:57:29.361+0000 7fb2224e8700 -1 mgr handle_command module 'orchestrator' command handler threw exception: 'partialmethod' object is not callable
2020-02-21T06:57:29.372 INFO:tasks.ceph.mgr.x.smithi071.stderr:2020-02-21T06:57:29.361+0000 7fb2224e8700 -1 mgr.server reply reply (22) Invalid argument TypeError: 'partialmethod' object is not callable
2020-02-21T06:57:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 22
2020-02-21T06:57:29.375 INFO:tasks.cephfs_test_runner:ERROR
2020-02-21T06:57:29.376 INFO:tasks.cephfs_test_runner:
2020-02-21T06:57:29.376 INFO:tasks.cephfs_test_runner:======================================================================
2020-02-21T06:57:29.376 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest)
2020-02-21T06:57:29.376 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/mgr/dashboard/test_host.py", line 21, in setUpClass
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: cls.mgr_cluster.mon_manager.raw_cluster_cmd(*cmd)
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/ceph_manager.py", line 1358, in raw_cluster_cmd
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: stdout=StringIO(),
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
2020-02-21T06:57:29.377 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: r.wait()
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner:CommandFailedError: Command failed on smithi071 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early orch set backend test_orchestrator'
2020-02-21T06:57:29.378 INFO:tasks.cephfs_test_runner:
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:Ran 67 tests in 697.850s
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:======================================================================
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest)
2020-02-21T06:57:29.379 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/mgr/dashboard/test_host.py", line 21, in setUpClass
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: cls.mgr_cluster.mon_manager.raw_cluster_cmd(*cmd)
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/ceph_manager.py", line 1358, in raw_cluster_cmd
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: stdout=StringIO(),
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-02-21T06:57:29.380 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner: r.wait()
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner:CommandFailedError: Command failed on smithi071 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early orch set backend test_orchestrator'
2020-02-21T06:57:29.381 INFO:tasks.cephfs_test_runner:
2020-02-21T06:57:29.381 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest)
2020-02-21T06:57:29.382 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2020-02-21T06:57:29.391 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2020-02-21T06:57:29.409 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 34, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/rgw.py", line 392, in task
yield
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-laura-testing-32552-32829-33178/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: setUpClass (tasks.mgr.dashboard.test_host.HostControllerTest)
2020-02-21T06:57:29.410 DEBUG:tasks.rgw.client.0:waiting for process to exit
</code></pre>
See:
* http://pulpito.ceph.com/laura-2020-02-21_06:19:27-rados:dashboard-wip-laura-testing-32552-32829-33178-distro-basic-smithi/
Related to https://github.com/ceph/ceph/pull/33432