Bug #61649
closedteuthology-lock fails with paramiko related errors
0%
Description
teuthology-lock fails to lock machines. Here is the output for an attempt to lock machines:
Command: teuthology-lock --os-type ubuntu --os-version 20.04 --lock-many 1 --machine-type smithi > targets.yaml
Output:
2023-06-12 14:13:59,670.670 INFO:teuthology.lock.ops:Start node 'smithi141.front.sepia.ceph.com' reimaging
2023-06-12 14:13:59,671.671 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: reset os type and version on server
2023-06-12 14:13:59,671.671 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2023-06-12 14:13:59,700.700 INFO:teuthology.lock.ops:Node 'smithi141.front.sepia.ceph.com' reimaging is complete
2023-06-12 14:13:59,872.872 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of ubuntu 20.04
2023-06-12 14:14:00,088.088 INFO:teuthology.orchestra.console:Power off smithi141
2023-06-12 14:14:12,565.565 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2023-06-12 14:14:12,666.666 INFO:teuthology.orchestra.console:Power on smithi141
2023-06-12 14:14:16,917.917 INFO:teuthology.orchestra.console:Power on for smithi141 completed
2023-06-12 14:14:17,018.018 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish
2023-06-12 14:18:21,424.424 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:30,512.512 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:39,600.600 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:48,688.688 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:20:37,491.491 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication timeout.
2023-06-12 14:20:38,524.524 ERROR:paramiko.transport:Unknown exception: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:Traceback (most recent call last):
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2163, in run
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: handler(self.auth_handler, m)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 375, in parse_service_accept
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: sig = self.private_key.sign_ssh_data(blob, algorithm)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 434, in sign_ssh_data
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: ptype, result = self.agent._send_message(msg)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 95, in _send_message
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: data = self._read_all(4)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 100, in _read_all
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: result = self._conn.recv(wanted)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 663, in recv
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: self._wait(self._read_event)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 297, in gevent._gevent_c_hub_primitives._primitive_wait
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:
Traceback (most recent call last):
File "/home/suriarte/src/teuthology/virtualenv/bin/teuthology-lock", line 8, in <module>
sys.exit(main())
File "/home/suriarte/src/teuthology/scripts/lock.py", line 18, in main
sys.exit(teuthology.lock.cli.main(parse_args(sys.argv[1:])))
File "/home/suriarte/src/teuthology/teuthology/lock/cli.py", line 210, in main
result = ops.lock_many(ctx, ctx.num_to_lock, ctx.machine_type, user,
File "/home/suriarte/src/teuthology/teuthology/lock/ops.py", line 142, in lock_many
return reimage_machines(ctx, machines, machine_type)
File "/home/suriarte/src/teuthology/teuthology/lock/ops.py", line 324, in reimage_machines
log.info("Node '%s' reimaging is complete", machine)
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 84, in exit
for result in self:
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 98, in next
resurrect_traceback(result)
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info1
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 23, in capture_traceback
return func(args, **kwargs)
File "/home/suriarte/src/teuthology/teuthology/provision/_init__.py", line 39, in reimage
return obj.create()
File "/home/suriarte/src/teuthology/teuthology/provision/fog.py", line 90, in create
self._wait_for_ready()
File "/home/suriarte/src/teuthology/teuthology/provision/fog.py", line 273, in _wait_for_ready
self.remote.connect()
File "/home/suriarte/src/teuthology/teuthology/orchestra/remote.py", line 386, in connect
self.ssh = connection.connect(*args)
File "/home/suriarte/src/teuthology/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
self._auth(
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 702, in _auth
self._transport.auth_publickey(username, key)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1634, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 244, in wait_for_response
raise e
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2163, in run
handler(self.auth_handler, m)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 375, in _parse_service_accept
sig = self.private_key.sign_ssh_data(blob, algorithm)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 434, in sign_ssh_data
ptype, result = self.agent._send_message(msg)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 95, in _send_message
data = self._read_all(4)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 100, in _read_all
result = self._conn.recv(wanted)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 663, in recv
self._wait(self._read_event)
File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 297, in gevent._gevent_c_hub_primitives._primitive_wait
gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>
Updated by Ali Maredia 11 months ago
So the error one of the RGW developers (Samarah) sent that opens up this BZ is not the one I was used to seeing. This is the error that another RGW Developer (Pritha) just hit that I've seen multiple times.
2023-06-13 04:55:52,607.607 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging
2023-06-13 04:55:52,607.607 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server
2023-06-13 04:55:52,607.607 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server
2023-06-13 04:55:52,821.821 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of ubuntu 22.04
2023-06-13 04:55:53,146.146 INFO:teuthology.orchestra.console:Power off smithi016
2023-06-13 04:55:53,287.287 INFO:teuthology.orchestra.console:Power off for smithi016 completed
2023-06-13 04:55:53,387.387 INFO:teuthology.orchestra.console:Power on smithi016
2023-06-13 04:55:57,641.641 INFO:teuthology.orchestra.console:Power on for smithi016 completed
2023-06-13 04:55:57,742.742 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish
2023-06-13 04:59:32,336.336 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-06-13 04:59:41,424.424 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-06-13 04:59:50,512.512 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-06-13 05:00:56,580.580 WARNING:teuthology.provision.fog:timed out
2023-06-13 05:01:05,616.616 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16
2023-06-13 05:01:11,685.685 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:17,710.710 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:23,735.735 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:29,761.761 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:35,786.786 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:41,812.812 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:47,837.837 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:53,862.862 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:01:59,888.888 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:05,914.914 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:11,939.939 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:17,966.966 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:23,970.970 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:Traceback (most recent call last):
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout)
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: buf += self._read_timeout(timeout)
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: raise EOFError()
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:EOFError
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:Traceback (most recent call last):
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: self._check_banner()
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport: raise SSHException(
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-06-13 05:02:24,032.032 ERROR:paramiko.transport:
2023-06-13 05:02:24,033.033 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-06-13 05:02:30,059.059 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:36,084.084 WARNING:teuthology.provision.fog:No authentication methods available
2023-06-13 05:02:42,087.087 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-06-13 05:02:42,087.087 ERROR:paramiko.transport:Traceback (most recent call last):
2023-06-13 05:02:42,087.087 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-06-13 05:02:42,087.087 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout)
2023-06-13 05:02:42,087.087 ERROR:paramiko.transport: File "/home/prsrivas/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
Updated by Zack Cerza 11 months ago
The error in comment #1 is caused by a lack of a private SSH key. I've made a PR to improve the error there: https://github.com/ceph/teuthology/pull/1855
The error in the description is something I've never seen before. How many times / on what other dates was that seen?
Updated by Samarah Uriarte 11 months ago
Zack Cerza wrote:
The error in comment #1 is caused by a lack of a private SSH key. I've made a PR to improve the error there: https://github.com/ceph/teuthology/pull/1855
The error in the description is something I've never seen before. How many times / on what other dates was that seen?
I have been experiencing this issue ever since I first tried to lock a machine a few months ago. Each retry has led to this outcome. As of today, I am still receiving the same error messages.