Bug #38757: mgr/ssh orchestrator doesn't work - Orchestrator - Ceph

Actions

Copy link

Bug #38757

closed

mgr/ssh orchestrator doesn't work

Added by 一帆师 about 5 years ago. Updated over 4 years ago.

Status:

Can't reproduce

Priority:

Normal

Assignee:

Noah Watkins

Category:

Target version:

Ceph - v14.0.0

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v14.0.0

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

I installed ceph-mgr-ssh-14.1.1 and python-remoto-0.0.35

set the ssh moudule by

"ceph mgr module enable ssh"

"ceph orchestrator set backend ssh"

then

I add the ssh_config_file as the follows:

Host node6
  HostName 192.168.10.3
  User root
  Port 22
  UserKnownHostsFile /root/.ssh/known_hosts
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /root/.ssh/id_rsa
  IdentitiesOnly yes
  LogLevel FATAL

it is the format as vagrant ssh-config,

"ceph ssh set-ssh-config -i ssh_config_file"

the I add node6 by

ceph orchestrator host add node6

then I tried to ls devices on node6.

the http://docs.ceph.com/docs/master/mgr/orchestrator_cli/ shows that

"host add, host ls, host rm ,mgr update, mon update,osd create,device ls" are already supported.

but it throws an error.

[root@node6 ~]# ceph orchestrator device ls --host=node6
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

the log output:

2019-03-15 05:05:11.351 7f0b6d009700  0 log_channel(audit) log [DBG] : from='client.74328 -' entity='client.admin' cmd=[{"prefix": "orchestrator device ls", "target": ["mgr", ""]}]: dispatch
2019-03-15 05:05:12.004 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v997: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:14.005 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v998: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:16.005 7f0b6c007700  0 log_channel(cluster) log [DBG] : pgmap v999: 44 pgs: 29 active+clean, 15 unknown; 1.6 KiB data, 82 MiB used, 3.0 TiB / 3.0 TiB avail
2019-03-15 05:05:16.356 7f0b6d80a700 -1 mgr handle_command module 'orchestrator_cli' command handler threw exception: expected 1 bytes, got 0
2019-03-15 05:05:16.356 7f0b6d80a700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

Files

Download all files

444444444.png (102 KB) 444444444.png		一帆师, 03/15/2019 09:08 AM
11111.png (4.9 KB) 11111.png		一帆师, 03/19/2019 12:45 AM

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nathan Cutler about 5 years ago

Project changed from Ceph to mgr
Category set to orchestrator

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Subject changed from ceph orchestrator doesn't work to mgr/ssh orchestrator doesn't work
Assignee set to Noah Watkins

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Description updated (diff)

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

https://github.com/pytest-dev/execnet/blob/c2a6b2c336e6c432ecbc8b415aa6238f7f9b813e/execnet/gateway_base.py#L379

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

一帆师, could you please verify that you can access node6 with ssh and no password? As you're running a Python 2 environment, Python might hide the real cause of the error, like in https://github.com/pytest-dev/execnet/issues/81

Actions

Copy link

Updated by Noah Watkins about 5 years ago

What distribution are you using?

Actions

Copy link

Updated by 一帆师 about 5 years ago

File 11111.png 11111.png added

Sebastian Wagner wrote:

一帆师, could you please verify that you can access node6 with ssh and no password? As you're running a Python 2 environment, Python might hide the real cause of the error, like in https://github.com/pytest-dev/execnet/issues/81

of course I can access it with no password,there is only one single ceph node named node6.

Actions

Copy link

Updated by 一帆师 about 5 years ago

Noah Watkins wrote:

What distribution are you using?

[root@node6 ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 
[root@node6 ~]# uname -a
Linux node6 5.0.1-1.el7.elrepo.x86_64 #1 SMP Sun Mar 10 10:09:55 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@node6 ~]# rpm -qa | grep ceph-mgr-ssh
ceph-mgr-ssh-14.1.1-0.el7.noarch
[root@node6 ~]# ceph --version
ceph version 14.1.1 (ae1b3cf8e2e29118c0682f0895e3f48a79895cbc) nautilus (rc)
[root@node6 ~]#

Actions

Copy link

Updated by 一帆师 about 5 years ago

Noah Watkins wrote:

What distribution are you using?

if I set ssh as the orchestrator backend,the dashboard of iscsi will throw an ERROR 500 "The server encountered an unexpected condition which prevented it from fulfilling the request." and lead the ceph-iscsi not work anymore.

Traceback (most recent call last):↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond↵    response.body = self.handler()↵  
File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__↵    self.body = self.oldhandler(*args, **kwargs)↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap↵    return self.newhandler(innerfunc, *args, **kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler↵    return handler(*args, **kwargs)↵  
File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__↵    return self.callable(*self.args, **self.kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 649, in inner↵    ret = func(*args, **kwargs)↵  
File "/usr/share/ceph/mgr/dashboard/controllers/iscsi.py", line 30, in status↵    if not IscsiGatewaysConfig.get_gateways_config()['gateways']:↵  
File "/usr/share/ceph/mgr/dashboard/services/iscsi_cli.py", line 85, in get_gateways_config↵    instances = OrchClient.instance().list_service_info("iscsi")↵  
File "/usr/share/ceph/mgr/dashboard/services/orchestrator.py", line 36, in list_service_info↵    completion = self._call("describe_service", service_type, None, None)↵  
File "/usr/share/ceph/mgr/dashboard/services/orchestrator.py", line 26, in _call↵    return mgr.remote(_backend, method, *args, **kwargs)↵  
File "/usr/share/ceph/mgr/mgr_module.py", line 1220, in remote↵    args, kwargs)↵RuntimeError: Remote method threw exception: Traceback (most recent call last):↵  
File "/usr/share/ceph/mgr/orchestrator.py", line 245, in describe_service↵    raise NotImplementedError()↵NotImplementedError.

Actions

Copy link

#10

Updated by Noah Watkins about 5 years ago

Thanks! I'm still not sure why list devices isn't working, but the issue with dashboard looks pretty straight forward to resolve. It looks like it just isn't handling unimplemented orchestrator functions gracefully.

Actions

Copy link

#11

Updated by Sebastian Wagner almost 5 years ago

[root@node6 ~]# ceph orchestrator device ls --host=node6
Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 862, in _handle_command
    return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
  File "/usr/share/ceph/mgr/mgr_module.py", line 337, in call
    return self.func(mgr, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 21, in inner
    return func(*args, **kwargs)
  File "/usr/share/ceph/mgr/orchestrator_cli/module.py", line 90, in _list_devices
    for inventory_node in completion.result:
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in result
    return list(map(lambda r: r.get(), self._result))
  File "/usr/share/ceph/mgr/ssh/module.py", line 37, in <lambda>
    return list(map(lambda r: r.get(), self._result))
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
EOFError: expected 1 bytes, got 0

Here, EOFError: expected 1 bytes, got 0 really has nothing to do with the traceback. https://github.com/ceph/ceph/pull/28213 will make the real Traceback appear in the log file.

Actions

Copy link

#12

Updated by Sebastian Wagner over 4 years ago

Status changed from New to Can't reproduce

we're reworking the ssh orchestrator for octopus. Closed.

Actions

Copy link

#13

Updated by Sage Weil over 4 years ago

Project changed from mgr to Orchestrator
Category deleted (~~orchestrator~~)

Actions

Copy link

#14

Updated by Sebastian Wagner over 3 years ago

Related to Bug #46990: execnet: EOFError: couldnt load message header, expected 9 bytes, got 0 added

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » Orchestrator

Custom queries

Bug #38757

mgr/ssh orchestrator doesn't work

Updated by Nathan Cutler about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Noah Watkins about 5 years ago

Updated by 一帆师 about 5 years ago

Updated by 一帆师 about 5 years ago

Updated by 一帆师 about 5 years ago

Updated by Noah Watkins about 5 years ago

Updated by Sebastian Wagner almost 5 years ago

Updated by Sebastian Wagner over 4 years ago

Updated by Sage Weil over 4 years ago

Updated by Sebastian Wagner over 3 years ago

Project

General

Profile

Ceph » Orchestrator

Custom queries

Bug #38757

mgr/ssh orchestrator doesn't work

Updated by Nathan Cutler about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Sebastian Wagner about 5 years ago

Updated by Noah Watkins about 5 years ago

Updated by 一帆 师 about 5 years ago

Updated by 一帆 师 about 5 years ago

Updated by 一帆 师 about 5 years ago

Updated by Noah Watkins about 5 years ago

Updated by Sebastian Wagner almost 5 years ago

Updated by Sebastian Wagner over 4 years ago

Updated by Sage Weil over 4 years ago

Updated by Sebastian Wagner over 3 years ago

Updated by 一帆师 about 5 years ago

Updated by 一帆师 about 5 years ago

Updated by 一帆师 about 5 years ago