Infrastructure

/a/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/7138859 - 1.1GB
/a/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/7138659 - 926M

In both of these examples, the teuthology.log files became so big that I couldn't view them with vim or less. This happened due to the traceback message repeatedly printing, i.e.:

2023-01-13T19:23:16.011 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/daemon/state.py", line 108, in signal
    self.proc.stdin.write(struct.pack('!b', sig))
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write
    self._write_all(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all
    count = self._write(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write
    self.channel.sendall(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall
    sent = self.send(s)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send
    return self._send(s, m)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send
    raise socket.error("Socket is closed")
OSError: Socket is closed
2023-01-13T19:23:16.069 INFO:teuthology.orchestra.run.smithi101.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2023-01-13T19:23:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 22
2023-01-13T19:23:16.073 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight
2023-01-13T19:23:16.112 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/daemon/state.py", line 108, in signal
    self.proc.stdin.write(struct.pack('!b', sig))
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write
    self._write_all(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all
    count = self._write(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write
    self.channel.sendall(data)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall
    sent = self.send(s)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send
    return self._send(s, m)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send
    raise socket.error("Socket is closed")
OSError: Socket is closed