Project

General

Profile

Actions
Copy link

Bug #65017

open

cephadm: log_channel(cephadm) log [ERR] : Failed to connect to smithi090 (10.0.0.9). Permission denied

Added by Prashant D about 1 month ago. Updated 2 days ago.

Status:
New
Priority:
Normal
Assignee:
Adam King
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

RADOS run on main branch : https://pulpito.ceph.com/teuthology-2024-03-10_20:46:54-rados-main-distro-default-smithi/

https://pulpito.ceph.com/teuthology-2024-03-10_20:46:54-rados-main-distro-default-smithi/7589772/

Observing failure for 'rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key}' test while adding host with ca signed ssh keys :

2024-03-10T22:04:22.556 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm...
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: cephadm 2024-03-10T22:04:21.315006+0000 mgr.a (mgr.14120) 2 : cephadm 1 [10/Mar/2024:22:04:21] ENGINE Bus STARTING
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: cephadm 2024-03-10T22:04:21.432290+0000 mgr.a (mgr.14120) 3 : cephadm 1 [10/Mar/2024:22:04:21] ENGINE Serving on https://10.0.0.9:7150
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: cephadm 2024-03-10T22:04:21.533557+0000 mgr.a (mgr.14120) 4 : cephadm 1 [10/Mar/2024:22:04:21] ENGINE Serving on http://10.0.0.9:8765
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: cephadm 2024-03-10T22:04:21.533716+0000 mgr.a (mgr.14120) 5 : cephadm 1 [10/Mar/2024:22:04:21] ENGINE Bus STARTED
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: audit 2024-03-10T22:04:21.773569+0000 mgr.a (mgr.14120) 6 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch
2024-03-10T22:04:23.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:22 smithi090 bash[21549]: audit 2024-03-10T22:04:21.787110+0000 mgr.a (mgr.14120) 7 : audit 0 from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch
2024-03-10T22:04:24.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:23 smithi090 bash[21549]: cluster 2024-03-10T22:04:22.842262+0000 mon.a (mon.0) 54 : cluster 0 mgrmap e8: a(active, since 2s)
2024-03-10T22:04:25.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:24 smithi090 bash[21549]: audit 2024-03-10T22:04:23.903788+0000 mgr.a (mgr.14120) 8 : audit 0 from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch
2024-03-10T22:04:25.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:24 smithi090 bash[21549]: audit 2024-03-10T22:04:23.912600+0000 mon.a (mon.0) 55 : audit 1 from='mgr.14120 10.0.0.9:0/189903570' entity='mgr.a'
2024-03-10T22:04:25.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:24 smithi090 bash[21549]: audit 2024-03-10T22:04:23.939030+0000 mon.a (mon.0) 56 : audit 0 from='mgr.14120 10.0.0.9:0/189903570' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
2024-03-10T22:04:26.778 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged
2024-03-10T22:04:26.778 INFO:teuthology.orchestra.run.smithi090.stdout:Using provided ssh private key and signed cert ...
2024-03-10T22:04:27.139 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:26 smithi090 bash[21549]: audit 2024-03-10T22:04:26.006458+0000 mgr.a (mgr.14120) 9 : audit 0 from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch
2024-03-10T22:04:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:29 smithi090 bash[21549]: audit 2024-03-10T22:04:28.225396+0000 mgr.a (mgr.14120) 10 : audit 0 from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm set-priv-key", "target": ["mon-mgr", ""]}]: dispatch
2024-03-10T22:04:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:29 smithi090 bash[21549]: audit 2024-03-10T22:04:28.230786+0000 mon.a (mon.0) 57 : audit 1 from='mgr.14120 10.0.0.9:0/189903570' entity='mgr.a'
2024-03-10T22:04:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:29 smithi090 bash[21549]: cephadm 2024-03-10T22:04:28.232276+0000 mgr.a (mgr.14120) 11 : cephadm 1 Set ssh ssh_identity_key
2024-03-10T22:04:29.389 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:29 smithi090 bash[21549]: cephadm 2024-03-10T22:04:28.232354+0000 mgr.a (mgr.14120) 12 : cephadm 1 Set ssh private key
2024-03-10T22:04:31.141 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090...
2024-03-10T22:04:31.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:31 smithi090 bash[21549]: audit 2024-03-10T22:04:30.355410+0000 mgr.a (mgr.14120) 13 : audit 0 from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm set-signed-cert", "target": ["mon-mgr", ""]}]: dispatch
2024-03-10T22:04:31.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:31 smithi090 bash[21549]: audit 2024-03-10T22:04:30.358855+0000 mon.a (mon.0) 58 : audit 1 from='mgr.14120 10.0.0.9:0/189903570' entity='mgr.a'
2024-03-10T22:04:31.639 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:31 smithi090 bash[21549]: cephadm 2024-03-10T22:04:30.360631+0000 mgr.a (mgr.14120) 14 : cephadm 1 Set ssh ssh_identity_cert
2024-03-10T22:04:32.803 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: debug 2024-03-10T22:04:32.607+0000 7f5a454cc700 -1 log_channel(cephadm) log [ERR] : Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:32.803 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Beginning auth for user root
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Auth failed for user root
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Aborting connection
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Traceback (most recent call last):
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     yield
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 102, in _remote_connection
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     preferred_auth=['publickey'], options=ssh_options)
2024-03-10T22:04:32.804 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     'Opening SSH connection to')
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     await conn.wait_established()
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     await self._waiter
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: asyncssh.misc.PermissionDenied: Permission denied
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: During handling of the above exception, another exception occurred:
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Traceback (most recent call last):
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1623, in _check_valid_addr
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     error_ok=True, no_fsid=True))
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     return self.event_loop.get_result(coro, timeout)
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     return future.result(timeout)
2024-03-10T22:04:32.805 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     return self.__get_result()
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     raise self._exception
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1553, in _run_cephadm
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     await self.mgr.ssh._remote_connection(host, addr)
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 108, in _remote_connection
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     raise
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/lib64/python3.6/contextlib.py", line 99, in __exit__
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     self.gen.throw(type, value, traceback)
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 135, in redirect_log
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     raise HostConnectionError(msg, host, addr)
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: cephadm.ssh.HostConnectionError: Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:32.806 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Beginning auth for user root
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Auth failed for user root
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Aborting connection
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: During handling of the above exception, another exception occurred:
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Traceback (most recent call last):
2024-03-10T22:04:32.807 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     return OrchResult(f(*args, **kwargs))
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 3293, in apply
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     results.append(self._apply(spec))
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 3043, in _apply
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     return self._add_host(cast(HostSpec, spec))
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1644, in _add_host
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     ip_addr = self._check_valid_addr(spec.hostname, spec.addr)
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1633, in _check_valid_addr
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]:     raise OrchestratorError(str(e))
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: orchestrator._interface.OrchestratorError: Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:32.808 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Beginning auth for user root
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Auth failed for user root
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Aborting connection
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: debug 2024-03-10T22:04:32.607+0000 7f5a454cc700 -1 mgr.server reply reply (22) Invalid argument Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Beginning auth for user root
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Auth failed for user root
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:32.809 INFO:journalctl@ceph.mgr.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21803]: [conn=0] Aborting connection
2024-03-10T22:04:33.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: audit 2024-03-10T22:04:32.486630+0000 mgr.a (mgr.14120) 15 : audit 0 from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "10.0.0.9", "target": ["mon-mgr", ""]}]: dispatch
2024-03-10T22:04:33.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: cephadm 2024-03-10T22:04:32.608865+0000 mgr.a (mgr.14120) 16 : cephadm 4 Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:33.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:33.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:33.140 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Beginning auth for user root
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Auth failed for user root
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Aborting connection
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Traceback (most recent call last):
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     yield
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 102, in _remote_connection
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     preferred_auth=['publickey'], options=ssh_options)
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     'Opening SSH connection to')
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect
2024-03-10T22:04:33.141 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     await conn.wait_established()
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in wait_established
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     await self._waiter
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: asyncssh.misc.PermissionDenied: Permission denied
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: During handling of the above exception, another exception occurred:
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Traceback (most recent call last):
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1623, in _check_valid_addr
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     error_ok=True, no_fsid=True))
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     return self.event_loop.get_result(coro, timeout)
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     return future.result(timeout)
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     return self.__get_result()
2024-03-10T22:04:33.142 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     raise self._exception
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1553, in _run_cephadm
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     await self.mgr.ssh._remote_connection(host, addr)
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 108, in _remote_connection
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     raise
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/lib64/python3.6/contextlib.py", line 99, in __exit__
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     self.gen.throw(type, value, traceback)
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 135, in redirect_log
2024-03-10T22:04:33.143 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     raise HostConnectionError(msg, host, addr)
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: cephadm.ssh.HostConnectionError: Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Beginning auth for user root
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Auth failed for user root
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Aborting connection
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: During handling of the above exception, another exception occurred:
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Traceback (most recent call last):
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper
2024-03-10T22:04:33.144 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     return OrchResult(f(*args, **kwargs))
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 3293, in apply
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     results.append(self._apply(spec))
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 3043, in _apply
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     return self._add_host(cast(HostSpec, spec))
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1644, in _add_host
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     ip_addr = self._check_valid_addr(spec.hostname, spec.addr)
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:   File "/usr/share/ceph/mgr/cephadm/module.py", line 1633, in _check_valid_addr
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]:     raise OrchestratorError(str(e))
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: orchestrator._interface.OrchestratorError: Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:33.145 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:33.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:33.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Beginning auth for user root
2024-03-10T22:04:33.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Auth failed for user root
2024-03-10T22:04:33.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Connection failure: Permission denied
2024-03-10T22:04:33.146 INFO:journalctl@ceph.mon.a.smithi090.stdout:Mar 10 22:04:32 smithi090 bash[21549]: [conn=0] Aborting connection
2024-03-10T22:04:33.486 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 22 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:cb2b48d76f56fc07bc6a177ed79951f825d2165e -e NODE_NAME=smithi090 -v /var/log/ceph/dfb9ff22-df29-11ee-95c7-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp35cj_t9a:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpwu_xl5zk:/etc/ceph/ceph.conf:z quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:cb2b48d76f56fc07bc6a177ed79951f825d2165e orch host add smithi090 10.0.0.9
2024-03-10T22:04:33.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr Error EINVAL: Failed to connect to smithi090 (10.0.0.9). Permission denied
2024-03-10T22:04:33.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr Log: Opening SSH connection to 10.0.0.9, port 22
2024-03-10T22:04:33.486 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0] Connected to SSH server at 10.0.0.9, port 22
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0]   Local address: 10.0.0.9, port 39190
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0]   Peer address: 10.0.0.9, port 22
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0] Beginning auth for user root
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0] Auth failed for user root
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0] Connection failure: Permission denied
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr [conn=0] Aborting connection
2024-03-10T22:04:33.487 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr

Actions
Copy link
 #1

Updated by Prashant D about 1 month ago

It seems to be mostly a infrastructure issue. Somehow test is failing to copy ca-key.pub to the smithi node.

The deployment steps involves CA signed keys generated and ca-key.pub copied to remote host
- Generate CA key

# sudo ssh-keygen -t rsa -f /root/ca-key -N ''

- Copy ca-key.pub to remote host (which we are trying to add it to the cluster)

  
# sudo scp /root/ca-key.pub root@$(hostname):/root/ca-key.pub

- Modify /etc/ssh/sshd_config and set path to trusted CA pub key

# vi /etc/ssh/sshd_config
# cat /etc/ssh/sshd_config | grep ca-key
TrustedUserCAKeys /root/ca-key.pub

- Make a new key pair

# sudo ssh-keygen -t rsa -f /root/cephadm-ssh-key -N ''
# sudo ssh-keygen -s /root/ca-key -I user_root -n root -V +52w /root/cephadm-ssh-key

- Set ssh_identity configs

# ceph_adm config-key set mgr/cephadm/ssh_identity_key -i /root/cephadm-ssh-key
# ceph_adm config-key set mgr/cephadm/ssh_identity_pub -i /root/cephadm-ssh-key.pub

Actions
Copy link
 #2

Updated by Prashant D about 1 month ago

Reproduce steps for vstart cluster :

$ git diff src/vstart.sh
diff --git a/src/vstart.sh b/src/vstart.sh
index dbb44cb434b..9fa65c33222 100755
--- a/src/vstart.sh
+++ b/src/vstart.sh
@@ -1342,11 +1342,29 @@ EOF
                | jq -r '.images[0].digest')
                ceph_adm config set global container_image "docker.io/ceph/daemon-base@$digest" 
        fi
-        ceph_adm config-key set mgr/cephadm/ssh_identity_key -i ~/.ssh/id_rsa
-        ceph_adm config-key set mgr/cephadm/ssh_identity_pub -i ~/.ssh/id_rsa.pub
+        #ceph_adm config-key set mgr/cephadm/ssh_identity_key -i ~/.ssh/id_rsa
+        #ceph_adm config-key set mgr/cephadm/ssh_identity_pub -i ~/.ssh/id_rsa.pub
         ceph_adm mgr module enable cephadm
         ceph_adm orch set backend cephadm
-        ceph_adm orch host add "$(hostname)" 
+        debug echo Setting up ca-signed-cert
+        # use auto generated cephadm-ssh-key
+        ls /root/ca-key && rm /root/ca-key
+        ls /root/ca-key.pub && rm /root/ca-key.pub
+        ls /root/cephadm-ssh-key && rm /root/cephadm-ssh-key
+        ls /root/cephadm-ssh-key.pub && rm /root/cephadm-ssh-key.pub
+        ls /root/cephadm-ssh-key-cert.pub && rm /root/cephadm-ssh-key-cert.pub
+        ls /root/tracker-65017.ca-key.pub && rm /root/tracker-65017.ca-key.pub
+        sudo ssh-keygen -t rsa -f /root/ca-key -N ''
+        # Reproducer step for tracker#65017, move ca-key.pub to assume that CA pub key has not been copied to remote host
+        mv /root/ca-key.pub /root/tracker-65017.ca-key.pub
+        grep -qxF 'TrustedUserCAKeys /root/ca-key.pub' /etc/ssh/sshd_config || echo 'TrustedUserCAKeys /root/ca-key.pub' >> /etc/ssh/sshd_config
+        sudo systemctl restart sshd
+        sudo ssh-keygen -t rsa -f /root/cephadm-ssh-key -N ''
+        sudo ssh-keygen -s /root/ca-key -I user_root -n root -V +52w /root/cephadm-ssh-key
+        ceph_adm config-key set mgr/cephadm/ssh_identity_key -i /root/cephadm-ssh-key
+        ceph_adm config-key set mgr/cephadm/ssh_identity_pub -i /root/cephadm-ssh-key.pub
+        ceph_adm cephadm set-signed-cert -i /root/cephadm-ssh-key-cert.pub
+        ceph_adm orch host add "$(hostname)" --addr 10.74.131.0
         ceph_adm orch apply crash '*'
         ceph_adm config set mgr mgr/cephadm/allow_ptrace true
     fi

vstart logs :

root@nodefrg:/home/user/ceph/build# MON=1 MGR=1 OSD=3 ../src/vstart.sh -n -d -x --cephadm
** going verbose **
rm -f core* 
hostname nodefrg
ip 10.0.0.33
port 40626
/home/user/ceph/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /home/user/ceph/build/keyring --cap mon 'allow *' 
creating /home/user/ceph/build/keyring
/home/user/ceph/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /home/user/ceph/build/keyring 
/home/user/ceph/build/bin/monmaptool --create --clobber --addv a [v2:10.0.0.33:40626,v1:10.0.0.33:40627] --print /tmp/ceph_monmap.3389577 
/home/user/ceph/build/bin/monmaptool: monmap file /tmp/ceph_monmap.3389577
/home/user/ceph/build/bin/monmaptool: generated fsid 169d905b-d3a0-4df8-866d-af9e77b44cc0
setting min_mon_release = quincy
epoch 0
fsid 169d905b-d3a0-4df8-866d-af9e77b44cc0
last_changed 2024-03-21T02:57:43.500526-0400
created 2024-03-21T02:57:43.500526-0400
min_mon_release 17 (quincy)
election_strategy: 1
0: [v2:10.0.0.33:40626/0,v1:10.0.0.33:40627/0] mon.a
/home/user/ceph/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.3389577 (1 monitors)
rm -rf -- /home/user/ceph/build/dev/mon.a 
mkdir -p /home/user/ceph/build/dev/mon.a 
/home/user/ceph/build/bin/ceph-mon --mkfs -c /home/user/ceph/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.3389577 --keyring=/home/user/ceph/build/keyring 
rm -- /tmp/ceph_monmap.3389577 
/home/user/ceph/build/bin/ceph-mon -i a -c /home/user/ceph/build/ceph.conf 
Populating config ...

[mgr]
    mgr/telemetry/enable = false
    mgr/telemetry/nag = false
Setting debug configs ...
Setting mon public_network ...
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config set mon public_network 10.74.128.0/22 
creating /home/user/ceph/build/dev/mgr.x/keyring
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring -i /home/user/ceph/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *' 
added key for mgr.x
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config set mgr mgr/dashboard/x/ssl_server_port 41626 --force 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config set mgr mgr/prometheus/x/server_port 9283 --force 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config set mgr mgr/restful/x/server_port 42626 --force 
Starting mgr.x
/home/user/ceph/build/bin/ceph-mgr -i x -c /home/user/ceph/build/ceph.conf 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
false
waiting for mgr to become available
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr stat 
true
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring -h 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring dashboard ac-user-create admin -i /home/user/ceph/build/dashboard-admin-secret.txt administrator --force-password 
{"username": "admin", "password": "$2b$12$ZD/n9UTVojyOWph/G0nHOuvPmzv95wo0R52dunu/VBboXsnAURBee", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711004275, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": false}
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring dashboard create-self-signed-cert 
Self-signed certificate created
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring osd pool create rbd 
pool 'rbd' created
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring osd pool application enable rbd rbd 
enabled application 'rbd' on pool 'rbd'
Enabling cephadm orchestrator
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config set global container_image docker.io/ceph/daemon-base@sha256:673ed6755ec01f8ba5ceb01598318bcd349375e136d16f7cde01b45e898822a5 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring mgr module enable cephadm 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring orch set backend cephadm 
Setting up ca-signed-cert
ls: cannot access '/root/ca-key': No such file or directory
/root/ca-key.pub
ls: cannot access '/root/cephadm-ssh-key': No such file or directory
ls: cannot access '/root/cephadm-ssh-key.pub': No such file or directory
ls: cannot access '/root/cephadm-ssh-key-cert.pub': No such file or directory
/root/tracker-65017.ca-key.pub
Generating public/private rsa key pair.
Your identification has been saved in /root/ca-key
Your public key has been saved in /root/ca-key.pub
The key fingerprint is:
SHA256:VrOUXL7VKy95KHg0kys8G4rcHKVFe7RLe2HyWdM+l1M root@nodefrg
The key's randomart image is:
+---[RSA 3072]----+
|            .    |
|         . +   . |
|         .*.. . .|
|        .ooooo  o|
|        S+.X.+ +E|
|       .= = X O.+|
|       o * * B *o|
|    . + o * o o +|
|     o + .       |
+----[SHA256]-----+
Generating public/private rsa key pair.
Your identification has been saved in /root/cephadm-ssh-key
Your public key has been saved in /root/cephadm-ssh-key.pub
The key fingerprint is:
SHA256:sttEsPfQd3ofRzGZkDjbnkweS3LLkOndJHTfYg71ldc root@nodefrg
The key's randomart image is:
+---[RSA 3072]----+
|            ... o|
|           o oooE|
|      .     B..*=|
|       o . *.Bo.*|
|      o S o &+X..|
|       = o o %.o |
|      . . . . ...|
|       +     . .o|
|      . .       .|
+----[SHA256]-----+
Signed user key /root/cephadm-ssh-key-cert.pub: id "user_root" serial 0 for root valid from 2024-03-21T02:57:00 to 2025-03-20T02:58:07
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config-key set mgr/cephadm/ssh_identity_key -i /root/cephadm-ssh-key 
set mgr/cephadm/ssh_identity_key
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring config-key set mgr/cephadm/ssh_identity_pub -i /root/cephadm-ssh-key.pub 
set mgr/cephadm/ssh_identity_pub
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring cephadm set-signed-cert -i /root/cephadm-ssh-key-cert.pub 
/home/user/ceph/build/bin/ceph -c /home/user/ceph/build/ceph.conf -k /home/user/ceph/build/keyring orch host add nodefrg --addr 10.0.0.0 
Error EINVAL: Failed to connect to nodefrg (10.0.0.0). Permission denied
Log: Opening SSH connection to 10.0.0.0, port 22
[conn=0] Connected to SSH server at 10.0.0.0, port 22
[conn=0]   Local address: 10.0.0.0, port 57948
[conn=0]   Peer address: 10.0.0.0, port 22
[conn=0] Beginning auth for user root
[conn=0] Auth failed for user root
[conn=0] Connection failure: Permission denied
[conn=0] Aborting connection

Actions
Copy link
 #3

Updated by Aishwarya Mathuria about 1 month ago

/a/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7609844

Actions
Copy link
 #4

Updated by Prashant D about 1 month ago

  • Assignee set to Adam King

Thanks Aishwarya.

From the /a/yuriw-2024-03-19_00:09:45-rados-wip-yuri5-testing-2024-03-18-1144-distro-default-smithi/7609844/teuthology.log, the ca-key.pub got copied to both smithi nodes :

2024-03-20T06:24:09.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo ssh-keygen -t rsa -f /root/ca-key -N ''
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:Generating public/private rsa key pair.
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:Your identification has been saved in /root/ca-key
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:Your public key has been saved in /root/ca-key.pub
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:The key fingerprint is:
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:SHA256:d4z9oYONGQrfcEFUibmv9Ab2G91zjWfC1oxYEgNS4KQ root@smithi187
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:The key's randomart image is:
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:+---[RSA 3072]----+
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:|        ++=+..   |
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:|       + oo..    |
2024-03-20T06:24:10.312 INFO:teuthology.orchestra.run.smithi187.stdout:|      E . ..o    |
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|          .= o   |
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|      . S =.= o  |
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|       o *+B.O B.|
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|        oo==* O.O|
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|          . ++ +o|
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:|           ...   |
2024-03-20T06:24:10.313 INFO:teuthology.orchestra.run.smithi187.stdout:+----[SHA256]-----+
2024-03-20T06:24:10.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo cat /root/ca-key.pub
2024-03-20T06:24:10.323 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:10.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:10.324 DEBUG:teuthology.orchestra.run.smithi187:> ' | sudo tee -a /etc/ssh/ca-key.pub
2024-03-20T06:24:10.377 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:10.378 INFO:teuthology.orchestra.run.smithi187.stdout:
2024-03-20T06:24:10.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo echo 'TrustedUserCAKeys /etc/ssh/ca-key.pub' | sudo tee -a /etc/ssh/sshd_config && sudo systemctl restart sshd
2024-03-20T06:24:10.430 INFO:teuthology.orchestra.run.smithi187.stdout:TrustedUserCAKeys /etc/ssh/ca-key.pub
2024-03-20T06:24:10.465 DEBUG:teuthology.orchestra.run.smithi195:> sudo echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:10.465 DEBUG:teuthology.orchestra.run.smithi195:> ' | sudo tee -a /etc/ssh/ca-key.pub
2024-03-20T06:24:10.479 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:10.480 INFO:teuthology.orchestra.run.smithi195.stdout:
2024-03-20T06:24:10.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo echo 'TrustedUserCAKeys /etc/ssh/ca-key.pub' | sudo tee -a /etc/ssh/sshd_config && sudo systemctl restart sshd
2024-03-20T06:24:10.535 INFO:teuthology.orchestra.run.smithi195.stdout:TrustedUserCAKeys /etc/ssh/ca-key.pub
2024-03-20T06:24:10.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo ssh-keygen -t rsa -f /root/cephadm-ssh-key -N '' && sudo ssh-keygen -s /root/ca-key -I user_root -n root -V +52w /root/cephadm-ssh-key
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:Generating public/private rsa key pair.
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:Your identification has been saved in /root/cephadm-ssh-key
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:Your public key has been saved in /root/cephadm-ssh-key.pub
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:The key fingerprint is:
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:SHA256:YvlTYrghwaMKugz/WROUUnt35qypfhJRegX1wt5TBWs root@smithi187
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:The key's randomart image is:
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:+---[RSA 3072]----+
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:|      .   .o. ...|
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:|   . . o  .... ..|
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:|    = + .o..= E .|
2024-03-20T06:24:11.219 INFO:teuthology.orchestra.run.smithi187.stdout:|   . = +o..* + . |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:|. . . B So. + o  |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:|o.   o B.o o   . |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:|+     + o.o      |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:|oo   o ..o.      |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:|....o  .oo       |
2024-03-20T06:24:11.220 INFO:teuthology.orchestra.run.smithi187.stdout:+----[SHA256]-----+
2024-03-20T06:24:11.230 INFO:teuthology.orchestra.run.smithi187.stderr:Signed user key /root/cephadm-ssh-key-cert.pub: id "user_root" serial 0 for root valid from 2024-03-20T06:23:00 to 2025-03-19T06:24:11
2024-03-20T06:24:11.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo cat /etc/ssh/ca-key.pub
2024-03-20T06:24:11.242 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:11.242 INFO:teuthology.orchestra.run.smithi187.stdout:
2024-03-20T06:24:11.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo cat /etc/ssh/sshd_config | grep TrustedUserCAKeys
2024-03-20T06:24:11.295 INFO:teuthology.orchestra.run.smithi187.stdout:TrustedUserCAKeys /etc/ssh/ca-key.pub
2024-03-20T06:24:11.296 DEBUG:teuthology.orchestra.run.smithi195:> sudo cat /etc/ssh/ca-key.pub
2024-03-20T06:24:11.308 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE2VLlFS5s83VXXsltcSP6Iw5MtoBPlZi8sekap2FzrZgyGSpF2sFxAL7ToDevwQwnhQv0N7M1Awcd25qpmSMe/zAiHXihHEHQ/T39OOBrkkYgLvVQI9hfK4AbJs0hNOCZIZP3M8Ek5tNaO6mlguEw1nJkUyYAGyatz8K1GYQncr5Lw5VW3QqNgJ1jd3Y25abZLMsYr0k8XhJlQb5Lom4Ak0R3dt+3SuhRnbtMsFgSeyYAFTFcQIXzt7TULwKto12raTHGGh5YzRHava3Q8OLZIdmlEk85P4pwsM/DIZ4MdA8DZGZnkyAXqfXNUJuNkayRK1aNCQiib08JhPWdXQICSx7jFy9zVl+u2tzVld4h+VW7kuEQL3KGFE8bIbBWZMxKApC4DvR6i8MHyaf78V8LWyTml/Euo/lEFJWzKCFYpW+GcRc9avBdvzp84H1JaZwmj07aFp5v/ZnaIwCmlLgAQO5eCWqHQ7uzi9CuuxqXNIepsSTGlQ7vGTl48qasr5M= root@smithi187
2024-03-20T06:24:11.308 INFO:teuthology.orchestra.run.smithi195.stdout:
2024-03-20T06:24:11.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo cat /etc/ssh/sshd_config | grep TrustedUserCAKeys
2024-03-20T06:24:11.364 INFO:teuthology.orchestra.run.smithi195.stdout:TrustedUserCAKeys /etc/ssh/ca-key.pub
2024-03-20T06:24:11.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo ls /root/
2024-03-20T06:24:11.376 INFO:teuthology.orchestra.run.smithi187.stdout:ca-key
2024-03-20T06:24:11.376 INFO:teuthology.orchestra.run.smithi187.stdout:ca-key.pub
2024-03-20T06:24:11.376 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-ssh-key
2024-03-20T06:24:11.377 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-ssh-key-cert.pub
2024-03-20T06:24:11.377 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-ssh-key.pub
2024-03-20T06:24:11.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph
2024-03-20T06:24:11.432 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph
2024-03-20T06:24:11.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph
2024-03-20T06:24:11.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph
....
....
2024-03-20T06:24:57.874 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:57 smithi187 bash[21552]: cephadm 2024-03-20T06:24:56.608365+0000 mgr.a (mgr.14120) 7 : cephadm [INF] [20/Mar/2024:06:24:56] ENGINE Bus STARTED
2024-03-20T06:24:57.874 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:57 smithi187 bash[21552]: cluster 2024-03-20T06:24:56.924321+0000 mon.a (mon.0) 54 : cluster [DBG] mgrmap e8: a(active, since 2s)
2024-03-20T06:24:57.875 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:57 smithi187 bash[21552]: audit 2024-03-20T06:24:57.590509+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14120 172.21.15.187:0/1986141585' entity='mgr.a'
2024-03-20T06:24:57.875 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:57 smithi187 bash[21552]: audit 2024-03-20T06:24:57.637462+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14120 172.21.15.187:0/1986141585' entity='mgr.a' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
2024-03-20T06:24:58.836 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:58 smithi187 bash[21552]: audit 2024-03-20T06:24:57.583813+0000 mgr.a (mgr.14120) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch
2024-03-20T06:24:59.976 INFO:teuthology.orchestra.run.smithi187.stdout:/usr/bin/ceph: stdout value unchanged
2024-03-20T06:24:59.977 INFO:teuthology.orchestra.run.smithi187.stdout:Using provided ssh private key and signed cert ...
2024-03-20T06:25:00.124 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:24:59 smithi187 bash[21552]: audit 2024-03-20T06:24:59.410553+0000 mgr.a (mgr.14120) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch
2024-03-20T06:25:02.527 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:02 smithi187 bash[21552]: audit 2024-03-20T06:25:01.239452+0000 mgr.a (mgr.14120) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm set-priv-key", "target": ["mon-mgr", ""]}]: dispatch
2024-03-20T06:25:02.527 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:02 smithi187 bash[21552]: audit 2024-03-20T06:25:01.244870+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14120 172.21.15.187:0/1986141585' entity='mgr.a'
2024-03-20T06:25:02.527 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:02 smithi187 bash[21552]: cephadm 2024-03-20T06:25:01.246466+0000 mgr.a (mgr.14120) 11 : cephadm [INF] Set ssh ssh_identity_key
2024-03-20T06:25:02.527 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:02 smithi187 bash[21552]: cephadm 2024-03-20T06:25:01.246551+0000 mgr.a (mgr.14120) 12 : cephadm [INF] Set ssh private key
2024-03-20T06:25:03.747 INFO:teuthology.orchestra.run.smithi187.stdout:Adding host smithi187...
2024-03-20T06:25:04.374 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:04 smithi187 bash[21552]: audit 2024-03-20T06:25:03.105213+0000 mgr.a (mgr.14120) 13 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm set-signed-cert", "target": ["mon-mgr", ""]}]: dispatch
2024-03-20T06:25:04.374 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:04 smithi187 bash[21552]: audit 2024-03-20T06:25:03.108756+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14120 172.21.15.187:0/1986141585' entity='mgr.a'
2024-03-20T06:25:04.374 INFO:journalctl@ceph.mon.a.smithi187.stdout:Mar 20 06:25:04 smithi187 bash[21552]: cephadm 2024-03-20T06:25:03.110143+0000 mgr.a (mgr.14120) 14 : cephadm [INF] Set ssh ssh_identity_cert
2024-03-20T06:25:05.283 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: debug 2024-03-20T06:25:05.102+0000 7fa49dfab700 -1 log_channel(cephadm) log [ERR] : Failed to connect to smithi187 (172.21.15.187). Permission denied
2024-03-20T06:25:05.283 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: Log: Opening SSH connection to 172.21.15.187, port 22
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0] Connected to SSH server at 172.21.15.187, port 22
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0]   Local address: 172.21.15.187, port 37514
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0]   Peer address: 172.21.15.187, port 22
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0] Beginning auth for user root
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0] Auth failed for user root
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0] Connection failure: Permission denied
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: [conn=0] Aborting connection
2024-03-20T06:25:05.284 INFO:journalctl@ceph.mgr.a.smithi187.stdout:Mar 20 06:25:05 smithi187 bash[21807]: Traceback (most recent call last):

So issue seems to be with either private cephadm-ssh-key or ca-signed-cert.

@Adam DC949 Any thoughts on this ?

Actions
Copy link
 #5

Updated by Nitzan Mordechai about 1 month ago

/a/yuriw-2024-03-25_20:36:27-rados-wip-yuri3-testing-2024-03-25-0744-distro-default-smithi/7621636

Actions
Copy link
 #6

Updated by Laura Flores about 1 month ago

/a/yuriw-2024-03-24_22:19:24-rados-wip-yuri10-testing-2024-03-24-1159-distro-default-smithi/7620502

Actions
Copy link
 #7

Updated by Laura Flores 29 days ago

  • Backport set to squid

/a/yuriw-2024-04-01_20:57:46-rados-wip-yuri3-testing-2024-04-01-0837-squid-distro-default-smithi/7634394

Actions
Copy link
 #8

Updated by Aishwarya Mathuria 16 days ago

/a/yuriw-2024-04-09_14:35:50-rados-wip-yuri5-testing-2024-03-21-0833-distro-default-smithi/7648574/

Actions
Copy link
 #9

Updated by Laura Flores 2 days ago

/a/yuriw-2024-04-20_15:32:38-rados-wip-yuriw-testing-20240419.185239-main-distro-default-smithi/7665003

Actions
Copy link

Also available in: Atom PDF