Ceph

If a user has a small to moderate size of root disk, they will probably have to modify the log rotation process for ceph, especially if they need to turn on debug. Our debug logs can generate a Gig of data in less than an hour.

Since many users have very large root disks, this will not apply to everyone, but we need to document how to change it for those that may hit this issue. If the logs fill the root disk, the system will crash, and the node will be unavailable.

the log rotation is kicked off by the daily cron job.

Steps needed on each node in the cluster:

• add a size limitation in the ceph log rotation file
• add a new cron job to check the log rotation more frequently

ceph's log rotation file is /etc/logrotate.d/ceph
The later command takes precedence, so you will want to add the size AFTER the timing.
default:
rotate 7
weekly
compress
sharedscripts
changed:
rotate 7
weekly
size 500M
compress
sharedscripts

then adding a cronjob to run the logrotate of ceph only would be needed, this example runs every 30minutes to check the log:
crontab -e
30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1