Project

General

Profile

Bug #57293

cephadm: logrotate conflict between ceph-common and cephadm logrotate configs

Added by Adam King over 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If you install ceph-common and run cephadm commands on the same host in certain distros (was able to recreate on ubuntu 20.04 but not centos 8.stream) two different logrotate configs end up created that conflict with each other

root@vm-00:~# cat /etc/logrotate.d/ceph-common 
/var/log/ceph/*.log {
    rotate 7
    daily
    compress
    sharedscripts
    postrotate
        killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror cephfs-mirror || pkill -1 -x "ceph-mon|ceph-mgr|ceph-mds|ceph-osd|ceph-fuse|radosgw|rbd-mirror|cephfs-mirror" || true
    endscript
    missingok
    notifempty
    su root ceph
}
root@vm-00:~# cat /etc/logrotate.d/cephadm
# created by cephadm
/var/log/ceph/cephadm.log {
    rotate 7
    daily
    compress
    missingok
    notifempty
    su root root
}

This causes a conflict that prevents log rotation from completing successfully

root@vm-00:~# logrotate /etc/logrotate.d/cephadm /etc/logrotate.d/ceph-common --force
error: /etc/logrotate.d/ceph-common:1 duplicate log entry for /var/log/ceph/cephadm.log

History

#1 Updated by Jan Horacek about 1 year ago

Hello, trying to solve this with finetuning glob in the ceph-common package owned logrotate config.
When i look at logfiles generated by ceph, `/var/log/ceph/ceph-*.log` looks like better glob, that does not cause "duplicate entry" error.

I could make PR on github for this change.

#2 Updated by Jan Horacek about 1 year ago

Other sollution mentioned on mailinglist, was keeping only empty file for the config pushed to host from ceph mgr - it should not be overwriten.

i found other problem with logrotate - /var/log/ceph/cephadm.log in containerized deployment is kept open even after logrotation, so this breaks correct rotation. the easiest fix is to add `delaycompress` and give the mgr process (all the ssh commands, for even monitoring, from active mgr nodes have time to close the `.1` file then and gzip could happily compress is the next day)

Also available in: Atom PDF