Bug #8314
closedAuthentication failed.
0%
Description
2014-05-08T01:36:06.758 DEBUG:teuthology.task.internal:connecting to ubuntu@plana35.front.sepia.ceph.com 2014-05-08T01:36:06.912 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 39, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 27, in run_one_task return fn(**kwargs) File "/home/teuthworker/teuthology-master/teuthology/task/internal.py", line 222, in connect remote.Remote(name=t, host_key=key, keep_alive=True, console=None)) File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 42, in __init__ self.ssh = ssh or self.connect() File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 47, in connect keep_alive=self.keep_alive) File "/home/teuthworker/teuthology-master/teuthology/orchestra/connection.py", line 89, in connect ssh.connect(**connect_args) File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 273, in connect self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys) File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 456, in _auth raise saved_exception AuthenticationException: Authentication failed.
these are popping up polluting the results. see for example
http://pulpito.ceph.com/teuthology-2014-05-07_02:30:12-rados-master-testing-basic-plana
Updated by Sage Weil almost 10 years ago
i see this at the same timestamp as the failures:
May 10 05:37:41 plana80 sshd[10708]: Connection closed by 10.214.137.23 [preauth] May 10 05:37:41 plana80 sshd[10710]: Connection closed by 10.214.137.23 [preauth] May 10 05:37:42 plana80 sshd[10712]: Accepted publickey for ubuntu from 10.214.137.23 port 35554 ssh2 May 10 05:37:42 plana80 sshd[10712]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0) May 10 05:37:42 plana80 sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/killall --quiet ceph-mon ceph-osd ceph-mds ceph-fuse ceph-disk radosgw ceph_test_rados rados apache2 May 10 05:37:42 plana80 sudo: pam_unix(sudo:session): session opened for user root by (uid=1000) May 10 05:37:42 plana80 sudo: pam_unix(sudo:session): session closed for user root May 10 05:40:16 plana80 sshd[1108]: Server listening on 0.0.0.0 port 22. May 10 05:40:16 plana80 sshd[1108]: Server listening on :: port 22. May 10 05:40:24 plana80 sshd[1526]: Accepted publickey for ubuntu from 10.214.137.23 port 36787 ssh2 May 10 05:40:24 plana80 sshd[1526]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
is that a nuke? and sshd restart (/reboot?) maybe?
Updated by Zack Cerza almost 10 years ago
I don't yet know why this is happening. But I think this is interesting:
2014-05-12T11:07:42.969 DEBUG:teuthology.task.internal:connecting to ubuntu@plana70.front.sepia.ceph.com 2014-05-12T11:07:43.086 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 39, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 27, in run_one_task return fn(**kwargs) File "/home/teuthworker/teuthology-master/teuthology/task/internal.py", line 222, in connect remote.Remote(name=t, host_key=key, keep_alive=True, console=None)) File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 43, in __init__ self._shortname = shortname File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 48, in connect File "/home/teuthworker/teuthology-master/teuthology/orchestra/connection.py", line 89, in connect ssh.connect(**connect_args) File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 273, in connect self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys) File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 456, in _auth raise saved_exception AuthenticationException: Authentication failed.
Now from plana70:
May 12 11:07:31 plana70 sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/chmod 0666 /tmp/tmpAVYByT May 12 11:07:31 plana70 sudo: pam_unix(sudo:session): session opened for user root by (uid=1000) May 12 11:07:31 plana70 sudo: pam_unix(sudo:session): session closed for user root May 12 11:07:31 plana70 sshd[3625]: subsystem request for sftp by user ubuntu May 12 11:07:31 plana70 sshd[3609]: pam_unix(sshd:session): session closed for user ubuntu May 12 11:07:36 plana70 sshd[8044]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8046]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8045]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8041]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8042]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8050]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8043]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8047]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8048]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:36 plana70 sshd[8049]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:43 plana70 sshd[8061]: Connection closed by 10.214.137.23 [preauth] May 12 11:07:44 plana70 sshd[8063]: Accepted publickey for ubuntu from 10.214.137.23 port 55036 ssh2 May 12 11:07:44 plana70 sshd[8063]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
So both before, and immediately after, the exception, auth had succeeded. Even from the same teuthology process.
Updated by Zack Cerza almost 10 years ago
- Status changed from 12 to 7
- Assignee set to Zack Cerza
This should fix the issue:
https://github.com/ceph/teuthology/commit/d945e564052ffc06ede59c78f99344f9fd3e571c
I'll leave this open so it doesn't get lost.
Updated by Zack Cerza almost 10 years ago
- Status changed from 7 to Resolved
I've seen some jobs raise AuthenticationException since I pushed the fix, and all of them so far have had to retry only once before reconnecting. I'm marking this resolved.
I also downgraded paramiko again because of #8327, so I kind of doubt we're even going to see this anymore. Double-resolved!