Infrastructure » sepia

As of right now, mira049's job history, most recent first:
running http://pulpito.front.sepia.ceph.com/sage-2016-02-12_05:26:12-rados-wip-sage-testing---basic-smithi/7142
dead http://pulpito.front.sepia.ceph.com/sage-2016-02-12_05:26:12-rados-wip-sage-testing---basic-smithi/7136

{'smithi041.front.sepia.ceph.com': {'invocation': {'module_name': 'apt', 'module_args': ''}, 'failed': True, 'parsed': False, 'msg': 'BECOME-SUCCESS-hcuscfuixbfpipjxokrryrevriuxxmfk\nOpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014\r\ndebug1: Reading configuration data /var/lib/teuthworker/.ssh/config\r\ndebug1: /var/lib/teuthworker/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for smithi*\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n File "<stdin>", line 2246, in <module>\n File "<stdin>", line 549, in main\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 107, in __init__\n self.open(progress)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.aHwvNE to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n'}, 'smithi049.front.sepia.ceph.com': {'invocation': {'module_name': 'apt', 'module_args': ''}, 'failed': True, 'parsed': False, 'item': 'ceph,ceph-common,libcephfs1,radosgw,python-ceph,python-rados,librbd1,librados2,openmpi-common', 'msg': 'BECOME-SUCCESS-zavvekjiqlacfpzehdvxtjpnsiwthyoe\nOpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014\r\ndebug1: Reading configuration data /var/lib/teuthworker/.ssh/config\r\ndebug1: /var/lib/teuthworker/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for smithi*\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: mux_client_request_session: master session id: 4\r\nTraceback (most recent call last):\n File "<stdin>", line 2246, in <module>\n File "<stdin>", line 549, in main\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 107, in __init__\n self.open(progress)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 153, in open\n self._records = apt_pkg.PackageRecords(self._cache)\nSystemError: E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_restricted_i18n_Translation-en - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-i386_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-i386_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-i386_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_main_binary-i386_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages - open (2: No such file or directory), E:Could not open file /var/lib/apt/lists/apt-mirror.front.sepia.ceph.com_archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages - open (2: No such file or directory)\n'}}

{'smithi049.front.sepia.ceph.com': 'SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh'}

SSH connection to smithi049 was lost: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=10.0.3-979-g35a4db4-1trusty ceph-mds=10.0.3-979-g35a4db4-1trusty ceph-common=10.0.3-979-g35a4db4-1trusty ceph-fuse=10.0.3-979-g35a4db4-1trusty ceph-test=10.0.3-979-g35a4db4-1trusty radosgw=10.0.3-979-g35a4db4-1trusty python-ceph=10.0.3-979-g35a4db4-1trusty libcephfs1=10.0.3-979-g35a4db4-1trusty libcephfs-java=10.0.3-979-g35a4db4-1trusty libcephfs-jni=10.0.3-979-g35a4db4-1trusty librados2=10.0.3-979-g35a4db4-1trusty librbd1=10.0.3-979-g35a4db4-1trusty rbd-fuse=10.0.3-979-g35a4db4-1trusty librados2=10.0.3-979-g35a4db4-1trusty librbd1=10.0.3-979-g35a4db4-1trusty'

SSH connection to smithi049 was lost: 'rm -rf -- /home/ubuntu/cephtest/data/test.client.0'

Job 7119 lost the ssh connection while dpkg was running. I have not investigated why.

teuthology.log:

2016-02-12T08:59:27.402 INFO:teuthology.orchestra.run.smithi049.stdout:Preparing to unpack .../libjs-jquery_1.7.2+dfsg-2ubuntu1_all.deb ...
2016-02-12T08:59:27.451 INFO:teuthology.orchestra.run.smithi049.stdout:Unpacking libjs-jquery (1.7.2+dfsg-2ubuntu1) ...
[...]
2016-02-12T09:01:16.836 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)

/var/log/syslog:

Feb 12 08:58:37 smithi049 dhclient: DHCPREQUEST of 172.21.15.49 on eth1 to 172.21.0.10 port 67 (xid=0x1a942231)
Feb 12 08:58:37 smithi049 dhclient: DHCPACK of 172.21.15.49 from 172.21.0.10
Feb 12 08:58:37 smithi049 dhclient: bound to 172.21.15.49 -- renewal in 3015 seconds.
Feb 12 09:01:41 smithi049 kernel: [21787.472823] init: tgt main process (6848) killed by KILL signal
Feb 12 09:02:22 smithi049 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="2820" x-info="http://www.rsyslog.com"] exiting on signal 15.
Feb 12 09:02:22 smithi049 rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="15604" x-info="http://www.rsyslog.com"] start

From http://qa-proxy.ceph.com/teuthology/sage-2016-02-12_05:26:12-rados-wip-sage-testing---basic-smithi/7136/teuthology.log:

2016-02-12T09:14:09.406 INFO:teuthology.task.ansible.out:TASK: [testnode | Update apt cache.] ****************************************** 

2016-02-12T09:14:18.075 INFO:teuthology.task.ansible.out:[0;31mfailed: [smithi041.front.sepia.ceph.com] => {"failed": true, "parsed": false}[0m
[0;31mBECOME-SUCCESS-hcuscfuixbfpipjxokrryrevriuxxmfk
2016-02-12T09:14:18.078 INFO:teuthology.task.ansible.out:
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014

debug1: Reading configuration data /var/lib/teuthworker/.ssh/config

debug1: /var/lib/teuthworker/.ssh/config line 1: Applying options for *

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: /etc/ssh/ssh_config line 19: Applying options for *

debug1: /etc/ssh/ssh_config line 56: Applying options for smithi*

debug1: auto-mux: Trying existing master

debug1: mux_client_request_session: master session id: 2

Traceback (most recent call last):
  File "<stdin>", line 2246, in <module>
  File "<stdin>", line 549, in main
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 107, in __init__
    self.open(progress)
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open
    self._cache = apt_pkg.Cache(progress)
SystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.aHwvNE to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems
[0m
[0;32mok: [smithi049.front.sepia.ceph.com] => {"changed": false}[0m

see also
smithi049 smithi041

smithi041 currently responds to apt-get update with

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

and dpkg -i shows libc-bin's state as 'it', which I'm assuming is the problem there, so it looks like dpkg was in fact interrupted in the past...but apt-get update doesn't fix it