Ceph » Orchestrator

This can lead to massive logs which are not compressed until the very end of the test:

2024-05-04T03:00:09.130 INFO:teuthology.orchestra.run.smithi081.stderr: 92.1% -- replaced with /var/log/ceph/2a8e1752-09bb-11ef-bc97-c7b262605968/ceph-osd.0.log.gz
2024-05-04T03:01:49.366 INFO:teuthology.orchestra.run.smithi081.stderr: 92.4% -- replaced with /var/log/ceph/2a8e1752-09bb-11ef-bc97-c7b262605968/ceph-osd.3.log.gz
2024-05-04T03:02:23.178 INFO:teuthology.orchestra.run.smithi081.stderr: 91.9% -- replaced with /var/log/ceph/2a8e1752-09bb-11ef-bc97-c7b262605968/ceph-osd.6.log.gz
2024-05-04T03:03:13.927 INFO:teuthology.orchestra.run.smithi081.stderr: 92.0% -- replaced with /var/log/ceph/2a8e1752-09bb-11ef-bc97-c7b262605968/ceph-osd.9.log.gz
2024-05-04T03:03:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:
2024-05-04T03:03:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:real     22m24.296s
2024-05-04T03:03:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:user     23m21.667s
2024-05-04T03:03:13.932 INFO:teuthology.orchestra.run.smithi081.stderr:sys      2m51.829s

See: /teuthology/pdonnell-2024-05-03_22:48:16-fs-wip-pdonnell-testing-20240503.163550-debug-distro-default-smithi/7688552/teuthology.log

It's important that logs be rotated regularly on a short time schedule and when the log is >1GB. Nodes CPU are often underutilized so it's very helpful during test cleanup if many logs were already rotated and compressed.

Also, I suggest having an infinite retention policy (perhaps change ceph.py too).