Tools » teuthology

i see this at the same timestamp as the failures:

May 10 05:37:41 plana80 sshd[10708]: Connection closed by 10.214.137.23 [preauth]
May 10 05:37:41 plana80 sshd[10710]: Connection closed by 10.214.137.23 [preauth]
May 10 05:37:42 plana80 sshd[10712]: Accepted publickey for ubuntu from 10.214.137.23 port 35554 ssh2
May 10 05:37:42 plana80 sshd[10712]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
May 10 05:37:42 plana80 sudo:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/killall --quiet ceph-mon ceph-osd ceph-mds ceph-fuse ceph-disk radosgw ceph_test_rados rados apache2
May 10 05:37:42 plana80 sudo: pam_unix(sudo:session): session opened for user root by (uid=1000)
May 10 05:37:42 plana80 sudo: pam_unix(sudo:session): session closed for user root
May 10 05:40:16 plana80 sshd[1108]: Server listening on 0.0.0.0 port 22.
May 10 05:40:16 plana80 sshd[1108]: Server listening on :: port 22.
May 10 05:40:24 plana80 sshd[1526]: Accepted publickey for ubuntu from 10.214.137.23 port 36787 ssh2
May 10 05:40:24 plana80 sshd[1526]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

is that a nuke? and sshd restart (/reboot?) maybe?

I don't yet know why this is happening. But I think this is interesting:

http://pulpito.front.sepia.ceph.com/teuthology-2014-05-12_02:30:08-rados-master-testing-basic-plana/250490/

2014-05-12T11:07:42.969 DEBUG:teuthology.task.internal:connecting to ubuntu@plana70.front.sepia.ceph.com
2014-05-12T11:07:43.086 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 39, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 27, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/teuthology-master/teuthology/task/internal.py", line 222, in connect
    remote.Remote(name=t, host_key=key, keep_alive=True, console=None))
  File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 43, in __init__
    self._shortname = shortname
  File "/home/teuthworker/teuthology-master/teuthology/orchestra/remote.py", line 48, in connect

  File "/home/teuthworker/teuthology-master/teuthology/orchestra/connection.py", line 89, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 273, in connect
    self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys)
  File "/home/teuthworker/teuthology-master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 456, in _auth
    raise saved_exception
AuthenticationException: Authentication failed.

Now from plana70:

May 12 11:07:31 plana70 sudo:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/chmod 0666 /tmp/tmpAVYByT
May 12 11:07:31 plana70 sudo: pam_unix(sudo:session): session opened for user root by (uid=1000)
May 12 11:07:31 plana70 sudo: pam_unix(sudo:session): session closed for user root
May 12 11:07:31 plana70 sshd[3625]: subsystem request for sftp by user ubuntu
May 12 11:07:31 plana70 sshd[3609]: pam_unix(sshd:session): session closed for user ubuntu
May 12 11:07:36 plana70 sshd[8044]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8046]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8045]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8041]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8042]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8050]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8043]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8047]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8048]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:36 plana70 sshd[8049]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:43 plana70 sshd[8061]: Connection closed by 10.214.137.23 [preauth]
May 12 11:07:44 plana70 sshd[8063]: Accepted publickey for ubuntu from 10.214.137.23 port 55036 ssh2
May 12 11:07:44 plana70 sshd[8063]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

So both before, and immediately after, the exception, auth had succeeded. Even from the same teuthology process.