Bug #48663: TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf' - mgr - Ceph

Actions

Copy link

Bug #48663

closed

TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'

Added by Neha Ojha over 3 years ago. Updated over 3 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2020-12-17T19:26:10.923 INFO:teuthology.orchestra.run.smithi063.stderr:2020-12-17T19:26:10.922+0000 7f9cec954700 20 mgrc start_command cmd: [{"prefix": "dashboard ac-user-create", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}]
2020-12-17T19:26:10.924 INFO:teuthology.orchestra.run.smithi063.stderr:2020-12-17T19:26:10.922+0000 7f9cec954700  1 -- 172.21.15.63:0/3352135212 --> [v2:172.21.15.63:6808/21153,v1:172.21.15.63:6809/21153] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "force_password": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce8057dc0 con 0x7f9ccc07f8d0
2020-12-17T19:26:10.925 INFO:tasks.ceph.mgr.x.smithi063.stderr:2020-12-17T19:26:10.926+0000 7f43e8f7a700 -1 mgr handle_command module 'dashboard' command handler threw exception: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2020-12-17T19:26:10.925 INFO:tasks.ceph.mgr.x.smithi063.stderr:2020-12-17T19:26:10.926+0000 7f43e8f7a700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
2020-12-17T19:26:10.926 INFO:tasks.ceph.mgr.x.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1211, in _handle_command
2020-12-17T19:26:10.926 INFO:tasks.ceph.mgr.x.smithi063.stderr:    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2020-12-17T19:26:10.926 INFO:tasks.ceph.mgr.x.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
2020-12-17T19:26:10.927 INFO:tasks.ceph.mgr.x.smithi063.stderr:    return self.func(mgr, **kwargs)
2020-12-17T19:26:10.927 INFO:tasks.ceph.mgr.x.smithi063.stderr:TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2020-12-17T19:26:10.928 INFO:tasks.ceph.mgr.x.smithi063.stderr:
2020-12-17T19:26:10.928 INFO:teuthology.orchestra.run.smithi063.stderr:2020-12-17T19:26:10.926+0000 7f9cdf7fe700  1 -- 172.21.15.63:0/3352135212 <== mgr.5480 v2:172.21.15.63:6808/21153 1 ==== mgr_command_reply(tid 0: -22 Traceback (most recent call last):
2020-12-17T19:26:10.928 INFO:teuthology.orchestra.run.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1211, in _handle_command
2020-12-17T19:26:10.929 INFO:teuthology.orchestra.run.smithi063.stderr:    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2020-12-17T19:26:10.929 INFO:teuthology.orchestra.run.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
2020-12-17T19:26:10.930 INFO:teuthology.orchestra.run.smithi063.stderr:    return self.func(mgr, **kwargs)
2020-12-17T19:26:10.930 INFO:teuthology.orchestra.run.smithi063.stderr:TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2020-12-17T19:26:10.930 INFO:teuthology.orchestra.run.smithi063.stderr:) v1 ==== 359+0+0 (secure 0 0 0) 0x7f9ce8057dc0 con 0x7f9ccc07f8d0
2020-12-17T19:26:10.931 INFO:teuthology.orchestra.run.smithi063.stderr:2020-12-17T19:26:10.926+0000 7f9cdf7fe700 20 mgrc handle_command_reply tid 0 r -22
2020-12-17T19:26:10.932 INFO:teuthology.orchestra.run.smithi063.stderr:Error EINVAL: Traceback (most recent call last):
2020-12-17T19:26:10.933 INFO:teuthology.orchestra.run.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1211, in _handle_command
2020-12-17T19:26:10.933 INFO:teuthology.orchestra.run.smithi063.stderr:    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2020-12-17T19:26:10.934 INFO:teuthology.orchestra.run.smithi063.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
2020-12-17T19:26:10.934 INFO:teuthology.orchestra.run.smithi063.stderr:    return self.func(mgr, **kwargs)
2020-12-17T19:26:10.935 INFO:teuthology.orchestra.run.smithi063.stderr:TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2020-12-17T19:26:10.935 INFO:teuthology.orchestra.run.smithi063.stderr:
2020-12-17T19:26:10.935 DEBUG:teuthology.orchestra.run:got remote process result: 22
2020-12-17T19:26:10.937 INFO:tasks.cephfs_test_runner:ERROR
2020-12-17T19:26:10.938 INFO:tasks.cephfs_test_runner:
2020-12-17T19:26:10.938 INFO:tasks.cephfs_test_runner:======================================================================
2020-12-17T19:26:10.939 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_auth.AuthTest)
2020-12-17T19:26:10.939 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-12-17T19:26:10.940 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-12-17T19:26:10.940 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/mgr/dashboard/helper.py", line 257, in setUpClass
2020-12-17T19:26:10.941 INFO:tasks.cephfs_test_runner:    cls.create_user('admin', 'admin', cls.AUTH_ROLES)
2020-12-17T19:26:10.941 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/mgr/dashboard/helper.py", line 124, in create_user
2020-12-17T19:26:10.942 INFO:tasks.cephfs_test_runner:    cls._ceph_cmd_with_secret(user_create_args, password)
2020-12-17T19:26:10.942 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/mgr/dashboard/helper.py", line 504, in _ceph_cmd_with_secret
2020-12-17T19:26:10.943 INFO:tasks.cephfs_test_runner:    return cls._ceph_cmd(cmd)
2020-12-17T19:26:10.943 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/mgr/dashboard/helper.py", line 488, in _ceph_cmd
2020-12-17T19:26:10.944 INFO:tasks.cephfs_test_runner:    res = cls.mgr_cluster.mon_manager.raw_cluster_cmd(*cmd)
2020-12-17T19:26:10.944 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1354, in raw_cluster_cmd
2020-12-17T19:26:10.945 INFO:tasks.cephfs_test_runner:    'stdout': StringIO()}).stdout.getvalue()
2020-12-17T19:26:10.946 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/ceph_manager.py", line 1347, in run_cluster_cmd
2020-12-17T19:26:10.946 INFO:tasks.cephfs_test_runner:    return self.controller.run(**kwargs)
2020-12-17T19:26:10.947 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 215, in run
2020-12-17T19:26:10.947 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-12-17T19:26:10.948 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 455, in run
2020-12-17T19:26:10.948 INFO:tasks.cephfs_test_runner:    r.wait()
2020-12-17T19:26:10.949 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 161, in wait
2020-12-17T19:26:10.949 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2020-12-17T19:26:10.950 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 183, in _raise_for_status
2020-12-17T19:26:10.950 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2020-12-17T19:26:10.951 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi063 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph dashboard ac-user-create admin --force-password -i /tmp/USVFW6HcqR2l8LqVLk6H'
2020-12-17T19:26:10.951 INFO:tasks.cephfs_test_runner:
2020-12-17T19:26:10.952 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-12-17T19:26:10.953 INFO:tasks.cephfs_test_runner:Ran 4 tests in 221.691s
2020-12-17T19:26:10.953 INFO:tasks.cephfs_test_runner:
2020-12-17T19:26:10.954 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2020-12-17T19:26:10.954 INFO:tasks.cephfs_test_runner:
2020-12-17T19:26:10.955 INFO:tasks.cephfs_test_runner:======================================================================
2020-12-17T19:26:10.955 INFO:tasks.cephfs_test_runner:ERROR: setUpClass (tasks.mgr.dashboard.test_auth.AuthTest)
2020-12-17T19:26:10.956 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------

/a/teuthology-2020-12-17_07:01:01-rados-master-distro-basic-smithi/5715104
/a/teuthology-2020-12-17_07:01:01-rados-master-distro-basic-smithi/5715124

Seems to be related to https://github.com/ceph/ceph/pull/38284/files#diff-09c82d2b566d42ede6e5d0990405da15594438e28e6ed47bb99f684d67ed6c43R726 but I cannot explain how, since this PR only merged later.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Neha Ojha over 3 years ago

Aha, suite_sha1: 7ef6453e4aff9bcef8bb2dd26182dd35bb98dee0 is different from ceph install sha1: 9b5210e497b0bf4322e569c37bf27a1c0d47f7dd. My guess is we won't see this particular bug later but we should address this suite/install sha1 discrepancy.

Actions

Copy link

Updated by Ernesto Puerta over 3 years ago

@Neha .: we have observed some failures in teuthology runs that seem to be caused by mismatching versions of packages vs. `qa/*` code, mostly when the PR has just been merged (next day usually).

For example, this other recently reported issue (https://tracker.ceph.com/issues/48636) happened in the code added the day before, and seemed to be due to QA tests executing "ceph --cluster ceph dashboard set-account-lockout-attempts 3" (a new command added) and receiving an exit code 22 (invalid argument). The only explanation for that failure is that the ceph-dashboard crashed... but other dashboard commands we run immediately before&after that one succeeded... Or simply it's that the given command doesn't exist in the packages under test.

Actions

Copy link