Bug #17125
closedteuthology-nuke fails when not running as ubuntu user
0%
Description
2016-08-24 15:25:47,122.122 INFO:teuthology.orchestra.run:Running command with timeout 30
2016-08-24 15:25:47,122.122 INFO:teuthology.orchestra.run.smithi011:Running: 'rm -f /home/ubuntu/.cephdeploy.conf'
2016-08-24 15:25:47,338.338 INFO:teuthology.orchestra.run.smithi011.stderr:rm: cannot remove ‘/home/ubuntu/.cephdeploy.conf’: Permission denied
2016-08-24 15:25:47,339.339 ERROR:teuthology.nuke:Could not nuke {'smithi011.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDskvOPgCDaXoVp9MBpKDGuyDlDK5RM61ccAzTx3rOEjPa6BE/O7oS0yqWb+bbxYlkRw79tA/YHgE92gGNIP8S+0adPD+LBoNOK8z48oKgNsM9IFTmavqMQo7DGH+ANkY4T10B2wFi10AxD1vxhxdS5GVwFcOofCXKfjQTAkIbu12U4Ja7F5xvOF/WzojlhEmycOetOdWXMQkEX61haY6lj1NjQs3O7wa8YJfr84DA5Mi8EBE/bH9w84pNvRZAIE3/JoSwCQZBxsC6gGNNjtR6+eo+/sMAb7zwLooVdo40Zs2aWoC1QYgHTZteUNeL0YhJaUMuzOojiheaqYmjI+HTF'}
Traceback (most recent call last):
File "/home/dgalloway/git/ceph/teuthology/teuthology/nuke/__init__.py", line 280, in nuke_one
nuke_helper(ctx, should_unlock)
File "/home/dgalloway/git/ceph/teuthology/teuthology/nuke/__init__.py", line 357, in nuke_helper
remove_configuration_files(ctx)
File "/home/dgalloway/git/ceph/teuthology/teuthology/nuke/actions.py", line 338, in remove_configuration_files
timeout=30
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/remote.py", line 192, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/run.py", line 402, in run
r.wait()
File "/home/dgalloway/git/ceph/teuthology/teuthology/orchestra/run.py", line 166, in wait
label=self.label)
CommandFailedError: Command failed on smithi011 with status 1: 'rm -f /home/ubuntu/.cephdeploy.conf'
2016-08-24 15:25:47,341.341 ERROR:teuthology.nuke:Could not nuke the following targets:
targets:
smithi011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDskvOPgCDaXoVp9MBpKDGuyDlDK5RM61ccAzTx3rOEjPa6BE/O7oS0yqWb+bbxYlkRw79tA/YHgE92gGNIP8S+0adPD+LBoNOK8z48oKgNsM9IFTmavqMQo7DGH+ANkY4T10B2wFi10AxD1vxhxdS5GVwFcOofCXKfjQTAkIbu12U4Ja7F5xvOF/WzojlhEmycOetOdWXMQkEX61haY6lj1NjQs3O7wa8YJfr84DA5Mi8EBE/bH9w84pNvRZAIE3/JoSwCQZBxsC6gGNNjtR6+eo+/sMAb7zwLooVdo40Zs2aWoC1QYgHTZteUNeL0YhJaUMuzOojiheaqYmjI+HTF
Just missing sudo here?
Updated by Patrick Donnelly over 7 years ago
David, I also encountered this problem but it was solved by simply logging in as ubuntu. (The problem in my case was that my ~/.ssh/config was setting my user to pdonnell which overrides what's in the yaml config. I just changed it to use ubuntu instead. I wonder if that's a bug itself...) Is there a reason nuke should work for other users?
Updated by Yuri Weinstein over 7 years ago
Patrick Donnelly wrote:
David, I also encountered this problem but it was solved by simply logging in as ubuntu. (The problem in my case was that my ~/.ssh/config was setting my user to pdonnell which overrides what's in the yaml config. I just changed it to use ubuntu instead. I wonder if that's a bug itself...) Is there a reason nuke should work for other users?
Just FYI I run t-nuke as yuriw all the time
Updated by Patrick Donnelly over 7 years ago
- Related to Bug #17237: teuthology does not use explicit username (ubuntu) from YAML file if ssh config specifies a username added
Updated by Zack Cerza over 7 years ago
- Assignee deleted (
Zack Cerza)
This might be a bug, but that doesn't mean I have time to work on it right now :-/
Updated by David Galloway over 7 years ago
- Priority changed from Normal to Urgent
This is still a (frequently-hit) problem
Updated by David Galloway over 7 years ago
- Related to Bug #17351: stale dirs left at the end of the nuke added
Updated by Dan Mick over 7 years ago
- Category set to Core
- Assignee set to Dan Mick
Updated by Dan Mick over 7 years ago
So, looking into this, I'm confused. I have a config with a specific vpm set to User dmick and the rest of them set to User ubuntu, like so:
Host vpm017
User dmick
Host vpm*
User ubuntu
ssh clearly uses 'dmick':
$ ssh vpm017 dmick@vpm017's password:
however, stopping in the connect task, after paramiko parses the configuration, here are 'opts' (from paramiko.SSHConfig().parse.lookup(host)):
{'forwardagent': 'yes',
'gssapiauthentication': 'no',
'hashknownhosts': 'no',
'hostname': 'vpm017.front.sepia.ceph.com',
'serveraliveinterval': '15',
'stricthostkeychecking': 'no',
'tcpkeepalive': 'no',
'user': 'ubuntu'}
So....paramiko doesn't parse .ssh/config the same way ssh does. O_o
Updated by Dan Mick over 7 years ago
ah...because the target name is ubuntu@<fqdn>, of course. Adding the full domain to my config file.
Updated by Dan Mick over 7 years ago
Yes, that made the difference, and moreover, vps is a bad test vehicle, because I don't get an account made on the vps (is that an oversight we should correct?) Testing with a mira.
Updated by Dan Mick over 7 years ago
...and the immediate failure is in mkdir /home/ubuntu/cephtest.
I think the right answer is to ignore User in the .ssh/config, and use the target-specified user.
Updated by Dan Mick over 7 years ago
Yeah. the more I think about it, the more I believe that the entire concept of overriding user or hostname is just madness. Just because you can do that, doesn't mean you should, and you can always edit your target list.
Updated by Dan Mick over 7 years ago
- Status changed from In Progress to Fix Under Review
Updated by Dan Mick over 7 years ago
https://github.com/ceph/teuthology/pull/964
97086c02349d118b30b2aacf1e3bfc4cdc1e5bc3
Updated by Dan Mick over 7 years ago
- Status changed from Fix Under Review to Resolved