Bug #42543
closedteuthology-lock: AttributeError: 'list' object has no attribute 'startswith'
0%
Description
Something changed recently preventing me from being able to lock machines. I'm running Ubuntu 18.04.
$ tl -v --lock-many 1 -m smithi --os-type centos --os-version 8.0
2019-10-29 16:05:07,751.751 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'dgalloway@p50', 'description': None, 'machine_type': 'smithi'}
2019-10-29 16:05:07,817.817 DEBUG:teuthology.lock.ops:locked ubuntu@smithi023.front.sepia.ceph.com
2019-10-29 16:05:08,211.211 INFO:teuthology.provision.fog.smithi023:Scheduling deploy of centos 8.0
2019-10-29 16:05:08,836.836 INFO:teuthology.orchestra.console:Power off smithi023
2019-10-29 16:05:08,837.837 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2019-10-29 16:05:10,165.165 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:14,457.457 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:18,757.757 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:23,867.867 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:24,242.242 INFO:teuthology.orchestra.console:Power off for smithi023 completed
2019-10-29 16:05:24,343.343 INFO:teuthology.orchestra.console:Power on smithi023
2019-10-29 16:05:24,344.344 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2019-10-29 16:05:24,618.618 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:28,908.908 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2019-10-29 16:05:29,279.279 INFO:teuthology.orchestra.console:Power on for smithi023 completed
2019-10-29 16:05:29,380.380 INFO:teuthology.provision.fog.smithi023:Waiting for deploy to finish
/home/dgalloway/git/ceph/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.ecdsa_curve.curve_class(), pointinfo
Traceback (most recent call last):
File "/home/dgalloway/git/ceph/teuthology/virtualenv/bin/teuthology-lock", line 11, in <module>
load_entry_point('teuthology', 'console_scripts', 'teuthology-lock')()
File "/home/dgalloway/git/ceph/teuthology/scripts/lock.py", line 18, in main
sys.exit(teuthology.lock.cli.main(parse_args(sys.argv[1:])))
File "/home/dgalloway/git/ceph/teuthology/teuthology/lock/cli.py", line 201, in main
ctx.desc, ctx.os_type, ctx.os_version, ctx.arch)
File "/home/dgalloway/git/ceph/teuthology/teuthology/lock/ops.py", line 145, in lock_many
reimaged[machine] = machines[machine]
File "/home/dgalloway/git/ceph/teuthology/teuthology/parallel.py", line 85, in __exit__
for result in self:
File "/home/dgalloway/git/ceph/teuthology/teuthology/parallel.py", line 99, in next
resurrect_traceback(result)
File "/home/dgalloway/git/ceph/teuthology/teuthology/parallel.py", line 22, in capture_traceback
return func(*args, **kwargs)
File "/home/dgalloway/git/ceph/teuthology/teuthology/provision/__init__.py", line 26, in reimage
return fog_obj.create()
File "/home/dgalloway/git/ceph/teuthology/teuthology/provision/fog.py", line 89, in create
self._wait_for_ready()
File "/home/dgalloway/git/ceph/teuthology/teuthology/provision/fog.py", line 263, in _wait_for_ready
self.remote.connect()
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/remote.py", line 69, in connect
self.ssh = connection.connect(**args)
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/connection.py", line 92, in connect
connect_args['key_filename'] = os.path.expanduser(key_filename)
File "/home/dgalloway/git/ceph/teuthology/virtualenv/lib/python2.7/posixpath.py", line 254, in expanduser
if not path.startswith('~'):
AttributeError: 'list' object has no attribute 'startswith'
Updated by David Galloway over 4 years ago
This commit broke teuthology-lock for me: https://github.com/ceph/teuthology/pull/1326/commits/5ac7c2f972e1e75d975ac95e9fa8ae13ef2f43f4
For debugging purposes, I added log.warn("key_filename: %s", key_filename) to see the value.
2019-11-04 16:50:12,100.100 WARNING:teuthology.orchestra.connection:key_filename: ['/home/dgalloway/.ssh/id_rsa_w541']
Updated by David Galloway over 4 years ago
I don't know enough about python to either understand what the original intent was or how to fix it.
Updated by Greg Farnum over 4 years ago
Hmm I pulled down the newest version and it's working for me on the teuthology box when I lock a rhel node.
It looks like you've somehow got a list of one element going down, which is opposite the claim of the commit. So we must have different callers giving it different kinds of things and need to fix up one side of those.
Updated by Kyrylo Shatskyy over 4 years ago
David Galloway wrote:
This commit broke
teuthology-lockfor me: https://github.com/ceph/teuthology/pull/1326/commits/5ac7c2f972e1e75d975ac95e9fa8ae13ef2f43f4For debugging purposes, I added
log.warn("key_filename: %s", key_filename)to see the value.[...]
Looks problem commit is https://github.com/ceph/teuthology/commit/5ac7c2f972e1e75d975ac95e9fa8ae13ef2f43f4
Updated by Kyrylo Shatskyy over 4 years ago
David Galloway wrote:
I don't know enough about python to either understand what the original intent was or how to fix it.
David, have you tried to set a separate virtualenv with fresh teuthology-lock setup?
Updated by Kyrylo Shatskyy over 4 years ago
hm... David, just wondering what is the content of your .ssh/config file, probably this can happen due to multiple identityfile in it?
Updated by Kyrylo Shatskyy over 4 years ago
Probable fix for this issue https://github.com/ceph/teuthology/pull/1344
Updated by Nathan Cutler over 4 years ago
- Status changed from New to Fix Under Review
- Assignee set to Kyrylo Shatskyy
Updated by David Galloway over 4 years ago
Kyrylo Shatskyy wrote:
David Galloway wrote:
I don't know enough about python to either understand what the original intent was or how to fix it.
David, have you tried to set a separate virtualenv with fresh teuthology-lock setup?
Yes, I rm -rfed my existing virtualenv and re-ran boostrap.
Updated by David Galloway over 4 years ago
Kyrylo Shatskyy wrote:
hm... David, just wondering what is the content of your .ssh/config file, probably this can happen due to multiple identityfile in it?
I think Zack said it was because a "list" of a single string was returned. But, yes, I do have multiple identityfiles offered by ssh-agent when I log in to machines.
Updated by Kyrylo Shatskyy over 4 years ago
- Status changed from Fix Under Review to Resolved