Bug #7072
closedlogrotate not sending HUP signal
0%
Description
I have a cluster where the Ceph Monitors are actually Virtual Machines.
They have relatively small 20GB root drives. According to the ceph hardware requirements 20GB should be enough.
Sometimes I get errors reported in zabbix that there is less than 20% disk space remaining.
I log onto a monitor and check the health. The status shows HEALTH_WARN.
In the past I had some wonky hardware and it was always /var/log where the culprit was.
On the actual machine, disk usage is at 85%. Yet there are no significantly large files sitting around.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 20511356 16420164 3026232 85% /
udev 2019768 4 2019764 1% /dev
tmpfs 817432 176 817256 1% /run
none 5120 0 5120 0% /run/lock
none 2043572 0 2043572 0% /run/shm
cd /var
sudo du -hsx * | sort -rh | head -10
195M lib
175M log
3.5M cache
740K backups
20K spool
4.0K tmp
4.0K opt
4.0K mail
4.0K local
0 run
When I restart ceph, the disk usage drops back down to a respectable level.
admin@ceph-mon2:/usr$ sudo service ceph-all restart
ceph-all stop/waiting
ceph-all start/running
admin@ceph-mon2:/usr$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda2 20511356 1096584 18349812 6% /
udev 2019768 4 2019764 1% /dev
tmpfs 817432 176 817256 1% /run
none 5120 0 5120 0% /run/lock
none 2043572 0 2043572 0% /run/shm
I've set my logrotate to be quite aggressive because of disk filling issues. But it would seem that perhaps the files are not being closed off properly?