Tools » teuthology

teuthology-lock fails to lock machines. Here is the output for an attempt to lock machines:

Command: teuthology-lock --os-type ubuntu --os-version 20.04 --lock-many 1 --machine-type smithi > targets.yaml

Output:
2023-06-12 14:13:59,670.670 INFO:teuthology.lock.ops:Start node 'smithi141.front.sepia.ceph.com' reimaging
2023-06-12 14:13:59,671.671 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: reset os type and version on server
2023-06-12 14:13:59,671.671 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2023-06-12 14:13:59,700.700 INFO:teuthology.lock.ops:Node 'smithi141.front.sepia.ceph.com' reimaging is complete
2023-06-12 14:13:59,872.872 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of ubuntu 20.04
2023-06-12 14:14:00,088.088 INFO:teuthology.orchestra.console:Power off smithi141
2023-06-12 14:14:12,565.565 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2023-06-12 14:14:12,666.666 INFO:teuthology.orchestra.console:Power on smithi141
2023-06-12 14:14:16,917.917 INFO:teuthology.orchestra.console:Power on for smithi141 completed
2023-06-12 14:14:17,018.018 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish
2023-06-12 14:18:21,424.424 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:30,512.512 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:39,600.600 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:18:48,688.688 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-06-12 14:20:37,491.491 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication timeout.
2023-06-12 14:20:38,524.524 ERROR:paramiko.transport:Unknown exception: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:Traceback (most recent call last):
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2163, in run
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: handler(self.auth_handler, m)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 375, in parse_service_accept
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: sig = self.private_key.sign_ssh_data(blob, algorithm)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 434, in sign_ssh_data
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: ptype, result = self.agent._send_message(msg)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 95, in _send_message
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: data = self._read_all(4)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 100, in _read_all
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: result = self._conn.recv(wanted)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 663, in recv
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: self._wait(self._read_event)
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport: File "src/gevent/_hub_primitives.py", line 297, in gevent._gevent_c_hub_primitives._primitive_wait
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>
2023-06-12 14:20:38,526.526 ERROR:paramiko.transport:
Traceback (most recent call last):
File "/home/suriarte/src/teuthology/virtualenv/bin/teuthology-lock", line 8, in <module>
sys.exit(main())
File "/home/suriarte/src/teuthology/scripts/lock.py", line 18, in main
sys.exit(teuthology.lock.cli.main(parse_args(sys.argv[1:])))
File "/home/suriarte/src/teuthology/teuthology/lock/cli.py", line 210, in main
result = ops.lock_many(ctx, ctx.num_to_lock, ctx.machine_type, user,
File "/home/suriarte/src/teuthology/teuthology/lock/ops.py", line 142, in lock_many
return reimage_machines(ctx, machines, machine_type)
File "/home/suriarte/src/teuthology/teuthology/lock/ops.py", line 324, in reimage_machines
log.info("Node '%s' reimaging is complete", machine)
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 84, in exit
for result in self:
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 98, in next
resurrect_traceback(result)
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info¹
File "/home/suriarte/src/teuthology/teuthology/parallel.py", line 23, in capture_traceback
return func(args, **kwargs)
File "/home/suriarte/src/teuthology/teuthology/provision/_init__.py", line 39, in reimage
return obj.create()
File "/home/suriarte/src/teuthology/teuthology/provision/fog.py", line 90, in create
self._wait_for_ready()
File "/home/suriarte/src/teuthology/teuthology/provision/fog.py", line 273, in _wait_for_ready
self.remote.connect()
File "/home/suriarte/src/teuthology/teuthology/orchestra/remote.py", line 386, in connect
self.ssh = connection.connect(*args)
File "/home/suriarte/src/teuthology/teuthology/orchestra/connection.py", line 106, in connect
ssh.connect(**connect_args)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
self._auth(
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 702, in _auth
self._transport.auth_publickey(username, key)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1634, in auth_publickey
return self.auth_handler.wait_for_response(my_event)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 244, in wait_for_response
raise e
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2163, in run
handler(self.auth_handler, m)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/auth_handler.py", line 375, in _parse_service_accept
sig = self.private_key.sign_ssh_data(blob, algorithm)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 434, in sign_ssh_data
ptype, result = self.agent._send_message(msg)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 95, in _send_message
data = self._read_all(4)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/paramiko/agent.py", line 100, in _read_all
result = self._conn.recv(wanted)
File "/home/suriarte/src/teuthology/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 663, in recv
self._wait(self._read_event)
File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 297, in gevent._gevent_c_hub_primitives._primitive_wait
gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7fd934b02bd0>>