Project

General

Profile

Bug #38757

mgr/ssh orchestrator doesn't work

Added by 一帆 师 3 months ago. Updated 26 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
orchestrator
Target version:
Start date:
03/15/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

I installed ceph-mgr-ssh-14.1.1 and python-remoto-0.0.35

set the ssh moudule by

"ceph mgr module enable ssh"

"ceph orchestrator set backend ssh"

then

I add the ssh_config_file as the follows:

Host node6
  HostName 192.168.10.3
  User root
  Port 22
  UserKnownHostsFile /root/.ssh/known_hosts
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /root/.ssh/id_rsa
  IdentitiesOnly yes
  LogLevel FATAL

it is the format as vagrant ssh-config,

"ceph ssh set-ssh-config -i ssh_config_file"

the I add node6 by

ceph orchestrator host add node6

then I tried to ls devices on node6.

the http://docs.ceph.com/docs/master/mgr/orchestrator_cli/ shows that

"host add, host ls, host rm ,mgr update, mon update,osd create,device ls" are already supported.

but it throws an error.

[root@node6 ~]# ceph orchestrator device ls --host=node6
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

the log output:

2019-03-15 05:05:11.351 7f0b6d009700  0 log_channel(audit) log [DBG] : from='client.74328 -' entity='client.admin' cmd=[{"prefix": "orchestrator device ls", "target": ["mgr", ""]}]: dispatch
2019-03-15 05:05:12.004 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v997: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:14.005 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v998: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:16.005 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v999: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:16.356 7f0b6d80a700 -1 mgr handle_command module 'orchestrator_cli' command handler threw exception: expected 1 bytes, got 0
2019-03-15 05:05:16.356 7f0b6d80a700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

444444444.png View (102 KB) 一帆 师, 03/15/2019 09:08 AM

11111.png View (4.9 KB) 一帆 师, 03/19/2019 12:45 AM

History

#1 Updated by Nathan Cutler 3 months ago

  • Project changed from Ceph to mgr
  • Category set to orchestrator

#2 Updated by Sebastian Wagner 3 months ago

  • Subject changed from ceph orchestrator doesn't work to mgr/ssh orchestrator doesn't work
  • Assignee set to Noah Watkins

#3 Updated by Sebastian Wagner 3 months ago

  • Description updated (diff)

#5 Updated by Sebastian Wagner 3 months ago

一帆 师, could you please verify that you can access node6 with ssh and no password? As you're running a Python 2 environment, Python might hide the real cause of the error, like in https://github.com/pytest-dev/execnet/issues/81

#6 Updated by Noah Watkins 3 months ago

What distribution are you using?

#7 Updated by 一帆 师 3 months ago

Sebastian Wagner wrote:

一帆 师, could you please verify that you can access node6 with ssh and no password? As you're running a Python 2 environment, Python might hide the real cause of the error, like in https://github.com/pytest-dev/execnet/issues/81

of course I can access it with no password,there is only one single ceph node named node6.

#8 Updated by 一帆 师 3 months ago

Noah Watkins wrote:

What distribution are you using?

[root@node6 ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 
[root@node6 ~]# uname -a
Linux node6 5.0.1-1.el7.elrepo.x86_64 #1 SMP Sun Mar 10 10:09:55 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@node6 ~]# rpm -qa | grep ceph-mgr-ssh
ceph-mgr-ssh-14.1.1-0.el7.noarch
[root@node6 ~]# ceph --version
ceph version 14.1.1 (ae1b3cf8e2e29118c0682f0895e3f48a79895cbc) nautilus (rc)
[root@node6 ~]# 

#9 Updated by 一帆 师 3 months ago

Noah Watkins wrote:

What distribution are you using?

if I set ssh as the orchestrator backend,the dashboard of iscsi will throw an ERROR 500 "The server encountered an unexpected condition which prevented it from fulfilling the request." and lead the ceph-iscsi not work anymore.

Traceback (most recent call last):↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond↵    response.body = self.handler()↵  
File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__↵    self.body = self.oldhandler(*args, **kwargs)↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap↵    return self.newhandler(innerfunc, *args, **kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler↵    return handler(*args, **kwargs)↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__↵    return self.callable(*self.args, **self.kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner↵    ret = func(*args, **kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 30, in status↵    if not IscsiGatewaysConfig.get_gateways_config()['gateways']:↵  
File "/usr/share/ceph/mgr/dashboard/services/iscsi_cli.py", line 85, in get_gateways_config↵    instances = OrchClient.instance().list_service_info("iscsi")↵  
File "/usr/share/ceph/mgr/dashboard/services/orchestrator.py", line 36, in list_service_info↵    completion = self._call("describe_service", service_type, None, None)↵  
File "/usr/share/ceph/mgr/dashboard/services/orchestrator.py", line 26, in _call↵    return mgr.remote(_backend, method, *args, **kwargs)↵  
File "/usr/share/ceph/mgr/mgr_module.py", line 1220, in remote↵    args, kwargs)↵RuntimeError: Remote method threw exception: Traceback (most recent call last):↵  
File "/usr/share/ceph/mgr/orchestrator.py", line 245, in describe_service↵    raise NotImplementedError()↵NotImplementedError.

#10 Updated by Noah Watkins 3 months ago

Thanks! I'm still not sure why list devices isn't working, but the issue with dashboard looks pretty straight forward to resolve. It looks like it just isn't handling unimplemented orchestrator functions gracefully.

#11 Updated by Sebastian Wagner 26 days ago

[root@node6 ~]# ceph orchestrator device ls --host=node6
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

Here, EOFError: expected 1 bytes, got 0 really has nothing to do with the traceback. https://github.com/ceph/ceph/pull/28213 will make the real Traceback appear in the log file.

Also available in: Atom PDF