Actions
Bug #58497
openteuthology.log files grow to ~1GB due to traceback repeatedly printing
Status:
New
Priority:
Normal
Assignee:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
/a/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/7138859 - 1.1GB
/a/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/7138659 - 926M
In both of these examples, the teuthology.log files became so big that I couldn't view them with vim or less. This happened due to the traceback message repeatedly printing, i.e.:
2023-01-13T19:23:16.011 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/daemon/state.py", line 108, in signal
self.proc.stdin.write(struct.pack('!b', sig))
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write
self._write_all(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all
count = self._write(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write
self.channel.sendall(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall
sent = self.send(s)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send
return self._send(s, m)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send
raise socket.error("Socket is closed")
OSError: Socket is closed
2023-01-13T19:23:16.069 INFO:teuthology.orchestra.run.smithi101.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2023-01-13T19:23:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 22
2023-01-13T19:23:16.073 DEBUG:teuthology.orchestra.run.smithi101:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight
2023-01-13T19:23:16.112 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/daemon/state.py", line 108, in signal
self.proc.stdin.write(struct.pack('!b', sig))
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 405, in write
self._write_all(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 523, in _write_all
count = self._write(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1364, in _write
self.channel.sendall(data)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 846, in sendall
sent = self.send(s)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 801, in send
return self._send(s, m)
File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1198, in _send
raise socket.error("Socket is closed")
OSError: Socket is closed
Updated by Laura Flores over 1 year ago
- Project changed from Ceph to Infrastructure
Updated by Kamoltat (Junior) Sirivadhna 10 months ago
/a/yuriw-2023-07-20_16:07:02-rados-wip-yuri2-testing-2023-07-19-1312-reef-distro-default-smithi/7345005/
Actions