Project

General

Profile

Actions
Copy link

Bug #49790

closed

"'dashboard' command handler threw exception: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'" in upgrade:octopus-x-pacific

Added by Yuri Weinstein about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Yuri Weinstein
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v15.2.9
ceph-qa-suite:
Pull request ID:
40156
Crash signature (v1):
Crash signature (v2):

Description

Run: https://pulpito.ceph.com/teuthology-2021-03-13_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/
Jobs: 104
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-13_14:22:03-upgrade:octopus-x-pacific-distro-basic-smithi/5962008/teuthology.log

2021-03-13T15:28:05.567 INFO:journalctl@ceph.mon.a.smithi095.stdout:Mar 13 15:28:05 smithi095 bash[13007]: audit 2021-03-13T15:28:03.998424+0000 mgr.y (mgr.14138) 3 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch
2021-03-13T15:28:05.568 INFO:journalctl@ceph.mon.a.smithi095.stdout:Mar 13 15:28:05 smithi095 bash[13007]: audit 2021-03-13T15:28:04.089569+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14138 172.21.15.95:0/1908081513' entity='mgr.y'
2021-03-13T15:28:05.568 INFO:journalctl@ceph.mon.a.smithi095.stdout:Mar 13 15:28:05 smithi095 bash[13007]: audit 2021-03-13T15:28:04.093643+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14138 172.21.15.95:0/1908081513' entity='mgr.y'
2021-03-13T15:28:06.147 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr Error EINVAL: Traceback (most recent call last):
2021-03-13T15:28:06.147 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/mgr_module.py", line 1179, in _handle_command
2021-03-13T15:28:06.148 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2021-03-13T15:28:06.148 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/mgr_module.py", line 318, in call
2021-03-13T15:28:06.148 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr     return self.func(mgr, **kwargs)
2021-03-13T15:28:06.148 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2021-03-13T15:28:06.148 INFO:teuthology.orchestra.run.smithi095.stderr:/usr/bin/ceph: stderr
2021-03-13T15:28:06.567 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]: debug 2021-03-13T15:28:06.138+0000 7f596fc8a700 -1 mgr handle_command module 'dashboard' command handler threw exception: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
2021-03-13T15:28:06.568 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]: debug 2021-03-13T15:28:06.142+0000 7f596fc8a700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
2021-03-13T15:28:06.568 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]:   File "/usr/share/ceph/mgr/mgr_module.py", line 1179, in _handle_command
2021-03-13T15:28:06.568 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]:     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2021-03-13T15:28:06.568 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]:   File "/usr/share/ceph/mgr/mgr_module.py", line 318, in call
2021-03-13T15:28:06.569 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]:     return self.func(mgr, **kwargs)
2021-03-13T15:28:06.569 INFO:journalctl@ceph.mgr.y.smithi095.stdout:Mar 13 15:28:06 smithi095 bash[13231]: TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'
Actions
Copy link
 #1

Updated by Sebastian Wagner about 3 years ago

  • Project changed from Ceph to mgr
Actions
Copy link
 #2

Updated by Alfonso Martínez about 3 years ago

  • Assignee set to Alfonso Martínez
Actions
Copy link
 #3

Updated by Ernesto Puerta about 3 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #5

Updated by Yuri Weinstein about 3 years ago

This issue seems also in 15.2.10 RC https://pulpito.ceph.com/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/ (got an unexpected keyword argument 'inbuf')

Actions
Copy link
 #6

Updated by Yuri Weinstein about 3 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #7

Updated by Alfonso Martínez about 3 years ago

  • Status changed from In Progress to Need More Info

From teuthology logs I see the QA run is using ceph image:
docker.io/ceph/daemon-base:latest-octopus

2021-03-15T18:07:08.504 INFO:tasks.workunit.client.0.smithi061.stderr:Non-zero exit code 22 from /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph -e CONTAINER_IMAGE=docker.io/ceph/daemon-base:latest-octopus -e NODE_NAME=smithi061 -v /var/log/ceph/00000000-0000-0000-0000-0000deadbeef:/var/log/ceph:z -v /tmp/ceph-tmpll1qnajg:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp0izxce5g:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpwri1z68g:/tmp/dashboard.pw:z docker.io/ceph/daemon-base:latest-octopus dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required
2021-03-15T18:07:08.504 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr Error EINVAL: Traceback (most recent call last):
2021-03-15T18:07:08.505 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/mgr_module.py", line 1179, in _handle_command
2021-03-15T18:07:08.505 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
2021-03-15T18:07:08.505 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr   File "/usr/share/ceph/mgr/mgr_module.py", line 318, in call
2021-03-15T18:07:08.505 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr     return self.func(mgr, **kwargs)
2021-03-15T18:07:08.506 INFO:tasks.workunit.client.0.smithi061.stderr:/usr/bin/ceph: stderr TypeError: ac_user_create_cmd() got an unexpected keyword argument 'inbuf'

I downloaded that image and is tagged as v15.2.9 so this container image DOES NOT contain the changes introduced in this merged PR:
https://github.com/ceph/ceph/pull/39436

// image latest-octopus that does not contain the changes:
# ceph -v
ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)

// This ceph version contains the change:
# ceph -v
ceph version 15.2.9-145-g4db20a73 (4db20a7386a7816fbf53ee7139cdb4ca22d9e1de) octopus (stable)

Actions
Copy link
 #8

Updated by Alfonso Martínez about 3 years ago

  • Affected Versions v15.2.9 added
  • Affected Versions deleted (v16.0.0)
Actions
Copy link
 #9

Updated by Ernesto Puerta about 3 years ago

Yuri: could you please confirm that the 15.2.10 RC includes the PR that Alfonso mentioned: git branch --contains b2ab238d4f074b84b61427ff16e47ad66ef501f8

It looks like it doesn't. If 15.2.10 becomes the official path to migrate to the first stable Pacific release, I'd suggest to include this PR, as otherwise it might complicate the migration:
  • That PR fixes a vulnerability and changes the way passwords and other sensitive information are fed into Ceph and Ceph Dashboard (from ceph dashboard <command> <password> to ceph dashboard <command> -i <path_to_password_file>.

In any case, this might also be the case that the upgrade QA task needs to be fixed accordingly to the new syntax.

Actions
Copy link
 #10

Updated by Yuri Weinstein about 3 years ago

Per IRC chat

in octopus-x tests:

(08:52:11 AM) joshd: yuriw: in the yaml file we have image: docker.io/ceph/daemon-base:latest-octopus
(08:52:33 AM) yuriw: so it takes whatever is 'latest' 
(08:52:34 AM) joshd: we need to use image: quay.ceph.io/ceph-ci/ceph:octopus
(08:52:37 AM) joshd: instead
Actions
Copy link
 #11

Updated by Alfonso Martínez about 3 years ago

  • Assignee changed from Alfonso Martínez to Yuri Weinstein
Actions
Copy link
 #12

Updated by Sage Weil about 3 years ago

  • Status changed from Need More Info to Resolved
Actions
Copy link
 #13

Updated by Alfonso Martínez about 3 years ago

  • Pull request ID set to 40156
Actions
Copy link

Also available in: Atom PDF