Bug #36524
Updated by Patrick Donnelly over 5 years ago
<pre>
2018-10-18T14:28:55.826 INFO:tasks.ceph:Shutting down logrotate
2018-10-18T14:28:55.826 INFO:teuthology.orchestra.run.smithi088:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2018-10-18T14:28:55.830 INFO:teuthology.orchestra.run.smithi178:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'
2018-10-18T14:28:55.938 INFO:teuthology.orchestra.run.smithi088:Running: 'sudo rm /etc/logrotate.d/ceph-test.conf'
2018-10-18T14:28:56.017 INFO:teuthology.orchestra.run.smithi178:Running: 'sudo rm /etc/logrotate.d/ceph-test.conf'
2018-10-18T14:28:56.116 INFO:tasks.ceph:Compressing logs...
2018-10-18T14:28:56.116 INFO:teuthology.orchestra.run.smithi088:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
2018-10-18T14:28:56.120 INFO:teuthology.orchestra.run.smithi178:Running: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"
2018-10-18T14:29:12.373 INFO:tasks.ceph:Archiving logs...
2018-10-18T14:29:12.375 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/pdonnell-2018-10-17_19:40:05-fs-wip-pdonnell-testing-20181017.175152-distro-basic-smithi/3152239/remote/smithi178/log
2018-10-18T14:29:12.376 INFO:teuthology.orchestra.run.smithi178:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
</pre>
From: /ceph/teuthology-archive/pdonnell-2018-10-17_19:40:05-fs-wip-pdonnell-testing-20181017.175152-distro-basic-smithi/3152239/teuthology.log
Teuthology branch: https://github.com/ceph/teuthology/commit/432e976e6965ad8fa466cf2360a1cd14efed0844
In another job where I killed gzip on the remote, I saw this traceback:
<pre>
2018-10-18T14:29:39.417 INFO:teuthology.orchestra.run.smithi082:Running: 'sudo tar cz -f - -C /var/log/ceph -- .'
2018-10-18T19:32:45.277 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2018-10-18T19:32:45.280 ERROR:teuthology.run_tasks:Manager failed: ceph
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/run_tasks.py", line 159, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20181017.175152/qa/tasks/ceph.py", line 1753, in task
'--no-mon-health-to-clog',
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/contextutil.py", line 46, in nested
if exit(*exc):
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20181017.175152/qa/tasks/ceph.py", line 243, in ceph_log
os.path.join(sub, 'log'))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/misc.py", line 771, in pull_directory
tar.makefile(ti, targetpath=os.path.join(localdir, sub))
File "/usr/lib/python2.7/tarfile.py", line 2233, in makefile
copyfileobj(source, target)
File "/usr/lib/python2.7/tarfile.py", line 266, in copyfileobj
shutil.copyfileobj(src, dst)
File "/usr/lib/python2.7/shutil.py", line 49, in copyfileobj
buf = fsrc.read(length)
File "/usr/lib/python2.7/tarfile.py", line 831, in read
buf += self.fileobj.read(size - len(buf))
File "/usr/lib/python2.7/tarfile.py", line 743, in read
return self.readnormal(size)
File "/usr/lib/python2.7/tarfile.py", line 758, in readnormal
return self.__read(size)
File "/usr/lib/python2.7/tarfile.py", line 750, in __read
raise ReadError("unexpected end of data")
ReadError: unexpected end of data
</pre>
From: /ceph/teuthology-archive/pdonnell-2018-10-17_19:40:05-fs-wip-pdonnell-testing-20181017.175152-distro-basic-smithi/3152202/teuthology.log
Edit: strace shows gzip is blocked on a write syscall:
<pre>
[root@smithi173 108181]# strace -f -v -p 108181 -s 100
strace: Process 108181 attached
write(1, "\31\32\317\214\2036\2371Ij\216\357\327b+\0317O\316,\f9 Dm\3z\204o\304\362\2348\0M-\r\303\272\206\322k\244\370\336\350\317`\34d\265\2\227*\271\273\370\305@S!(\332\326\267\357E>\302f\2732\246\35 \365\23\301B\343\1\253\312?\323\325\311Y\224Zf\370\350\301\36Y\217\370H\217"..., 16384^Cstrace: Process 108181 detached
</pre>