Project

General

Profile

Bug #15569

Two logrotate scripts

Added by Tim Bishop almost 8 years ago. Updated almost 7 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After upgrading from infernalis to jewel I have two logrotate scripts:

# dpkg -S ceph ceph-base | grep logrotate
ceph-base: /etc/logrotate.d/ceph.logrotate
ceph: /etc/logrotate.d/ceph

# dpkg -l ceph ceph-base
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  ceph           10.2.0-1trus amd64        distributed storage and file syst
ii  ceph-base      10.2.0-1trus amd64        common ceph daemon libraries and

# diff -u /etc/logrotate.d/ceph*
--- /etc/logrotate.d/ceph       2015-11-10 12:10:47.000000000 +0000
+++ /etc/logrotate.d/ceph.logrotate     2016-04-20 17:54:24.000000000 +0100
@@ -4,7 +4,7 @@
     compress
     sharedscripts
     postrotate
-        killall -q -1 ceph-mon ceph-mds ceph-osd radosgw || true
+        killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true
     endscript
     missingok
     notifempty

This results in cron errors:

/etc/cron.daily/logrotate:
error: ceph.logrotate:1 duplicate log entry for /var/log/ceph/ceph.audit.log

Looking at the package contents for the 10.2.0 ceph package it's not listed:

# dpkg -c /var/cache/apt/archives/ceph_10.2.0-1trusty_amd64.deb
drwxr-xr-x root/root         0 2016-04-20 17:57 ./
drwxr-xr-x root/root         0 2016-04-20 17:51 ./usr/
drwxr-xr-x root/root         0 2016-04-20 17:51 ./usr/share/
drwxr-xr-x root/root         0 2016-04-20 17:51 ./usr/share/doc/
drwxr-xr-x root/root         0 2016-04-20 17:57 ./usr/share/doc/ceph/
-rw-r--r-- root/root    984022 2016-04-20 17:01 ./usr/share/doc/ceph/ChangeLog.gz
-rw-r--r-- root/root      6087 2016-04-20 17:08 ./usr/share/doc/ceph/copyright
-rw-r--r-- root/root      2441 2016-04-20 17:08 ./usr/share/doc/ceph/changelog.Debian.gz

So this is just caused by the upgrade. I wonder if it affects any other similar files?

History

#1 Updated by Victor Mora over 7 years ago

We have the same problem at our jewel ceph with ubuntu trusty distribution:

  1. dpkg -l |grep ceph
    ii ceph 10.2.3-1trusty amd64 distributed storage and file system
    ii ceph-base 10.2.3-1trusty amd64 common ceph daemon libraries and management tools
    ...
  1. dpkg -S ceph ceph-base|grep logrotate
    ceph-base: /etc/logrotate.d/ceph.logrotate
    ceph: /etc/logrotate.d/ceph
The file differences are considerable in 10.2.3-1trusty:
  1. diff ceph ceph.logrotate
    7,26c7
    < if which service > /dev/null 2>&1 && [ x `which service` ]; then
    < service ceph reload >/dev/null
    < elif which invoke-rc.d > /dev/null 2>&1 && [ -x `which invoke-rc.d` ]; then
    < invoke-rc.d ceph reload >/dev/null
    < fi
    < # Possibly reload twice, but depending on ceph.conf the reload above may be a no-op
    < if which initctl > /dev/null 2>&1 && [ -x `which initctl` ]; then
    < for daemon in osd mon mds ; do
    < find -L /var/lib/ceph/$daemon/ -mindepth 1 -maxdepth 1 -regextype posix-egrep -regex '.*/[A-Za-z0-9]+
    [A-Za-z0-9._-]+' printf '%P\n' \
    < | while read f; do
    < if [ -e "/var/lib/ceph/$daemon/$f/done" -o -e "/var/lib/ceph/$daemon/$f/ready" ] && [ -e "/var/lib/ceph/$daemon/$f/upstart" ] && [ ! -e "/var/lib/ceph/$daemon/$f/sysvinit" ]; then
    < cluster="${f%%
    *}"
    < id="${f#*-}"
    <
    < initctl reload ceph-$daemon cluster="$cluster" id="$id" 2>/dev/null || :
    < fi
    < done
    < done
    < fi
    < killall q -1 ceph-fuse || true
    --

killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true

29a11

su ceph ceph

In xenial distribution, the only logrotate file is ceph.logrotate of the ceph-base package.

#2 Updated by Fabian Grünbichler over 7 years ago

this issue also exists when upgrading from Hammer to the 10.2.5 Debian Jessie packages (from the ceph.com repository). it seems like the "ceph" is now completely split up into the ceph-X packages, since the former only contains copyright information:

# dpkg -L ceph
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/ceph
/usr/share/doc/ceph/copyright
/usr/share/doc/ceph/ChangeLog.gz
/usr/share/doc/ceph/changelog.Debian.gz

renaming config files like this with no real reason is a bad idea because it's really ugly to clean up now..

#3 Updated by Greg Farnum almost 7 years ago

  • Status changed from New to Closed

Presumably everybody's hit this bug by now so it's too late to fix. :/

#4 Updated by Christian Zunker almost 7 years ago

Please reopen the bug.
We still see the problem in a fresh install, just with other package names:


dpkg -l | grep ceph
ii ceph 10.2.7-1xenial amd64 distributed storage and file system
ii ceph-base 10.2.7-1xenial amd64 common ceph daemon libraries and management tools
ii ceph-common 10.2.7-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fs-common 10.2.7-1xenial amd64 common utilities to mount and interact with a ceph file system
ii ceph-fuse 10.2.7-1xenial amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 10.2.7-1xenial amd64 metadata server for the ceph distributed file system
ii ceph-mon 10.2.7-1xenial amd64 monitor server for the ceph storage system
ii ceph-osd 10.2.7-1xenial amd64 OSD server for the ceph storage system
ii libcephfs1 10.2.7-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.7-1xenial amd64 Python libraries for the Ceph libcephfs library


dpkg -S ceph ceph-base ceph-common ceph-fs-common ceph-fuse ceph-mds ceph-mon ceph-osd | grep logrotate
ceph-base: /etc/logrotate.d/ceph.logrotate
ceph-common: /etc/logrotate.d/ceph-common

Both files have the same content:


/etc/logrotate.d > cat ceph-common ceph.logrotate
/var/log/ceph/*.log {
rotate 7
daily
compress
sharedscripts
postrotate
killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true
endscript
missingok
notifempty
su root ceph
}
/var/log/ceph/*.log {
rotate 7
daily
compress
sharedscripts
postrotate
killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true
endscript
missingok
notifempty
su root ceph
}

#5 Updated by Nathan Cutler almost 7 years ago

  • Status changed from Closed to New

What does "ceph --version" say? Where did you get these packages, exactly?

I looked at the debian packaging for 10.2.7 and can't find any trace of /etc/logrotate.d/ceph-common (not to say it's not happening as you describe, just that it's a mystery)

#6 Updated by Christian Zunker almost 7 years ago

@
  1. ceph --version
    ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)
    @

The packages are installed from this source: http://download.ceph.com/debian-jewel xenial

In case it helps: Installation was done with ceph-ansible

#7 Updated by Nathan Cutler almost 7 years ago

  • Status changed from New to Pending Backport
  • Assignee set to Kefu Chai
  • Backport set to jewel

The problem might be caused by this line [1] and, more generally, by not sufficiently reflecting the split of the former "ceph" megapackage into ceph-base, ceph-osd, etc.

It's fixed in master, so we'll backport the relevant bits to jewel. Thanks!

[1] https://github.com/ceph/ceph/blob/v10.2.7/debian/rules#L56

#9 Updated by Nathan Cutler almost 7 years ago

https://github.com/ceph/ceph/pull/9995/commits/457ddea9c8b0f000476798e99ee0aaeb003402c9

Also needs the jewel backport of #19390 (both backports should be done in a single PR)

Strike that. The changes are too intrusive for backporting. Will take a minimalistic approach.

#10 Updated by Nathan Cutler almost 7 years ago

  • Project changed from Ceph to ceph-ansible
  • Status changed from Pending Backport to New
  • Assignee deleted (Kefu Chai)
  • Priority changed from Normal to High
  • Backport deleted (jewel)

Christian, this is odd. I downloaded the ceph-base and ceph-common deb files from http://download.ceph.com/debian-jewel and clearly the ceph-common deb file does not include any logrotate file. Try it for yourself:

wget http://download.ceph.com/debian-jewel/pool/main/c/ceph/ceph-base_10.2.7-1xenial_amd64.deb
wget http://download.ceph.com/debian-jewel/pool/main/c/ceph/ceph-common_10.2.7-1xenial_amd64.deb
dpkg-deb --contents ceph-base_10.2.7-1xenial_amd64.deb | grep logrotate
dpkg-deb --contents ceph-common_10.2.7-1xenial_amd64.deb | grep logrotate

Here is the output:

$ dpkg-deb --contents ceph-base_10.2.7-1xenial_amd64.deb | grep logrotate
drwxr-xr-x root/root         0 2017-04-10 12:23 ./etc/logrotate.d/
-rw-r--r-- root/root       228 2017-04-10 12:23 ./etc/logrotate.d/ceph.logrotate
$ dpkg-deb --contents ceph-common_10.2.7-1xenial_amd64.deb | grep logrotate
$

I'm beginning to suspect that ceph-ansible is the culprit here. Moving bug to the ceph-ansible project.

#11 Updated by Alex Hart almost 7 years ago

I'm seeing this with ceph 12.0.2-pve1 (Proxmox VE 5.0 beta2) as well:

# dpkg -S ceph | grep logrotate
ceph-common: /etc/logrotate.d/ceph-common
ceph-base: /etc/logrotate.d/ceph
# diff -u /etc/logrotate.d/ceph-common /etc/logrotate.d/ceph
--- /etc/logrotate.d/ceph-common        2016-12-09 12:09:00.000000000 -0800
+++ /etc/logrotate.d/ceph       2017-04-26 01:20:20.000000000 -0700
@@ -4,7 +4,7 @@
     compress
     sharedscripts
     postrotate
-        killall -q -1 ceph-mon ceph-mds ceph-osd ceph-fuse radosgw || true
+        killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw || true
     endscript
     missingok
     notifempty

Which results in:

/etc/cron.daily/logrotate:
error: ceph-common:1 duplicate log entry for /var/log/ceph/ceph.audit.log

#12 Updated by Fabian Grünbichler almost 7 years ago

Alex Hart wrote:

I'm seeing this with ceph 12.0.2-pve1 (Proxmox VE 5.0 beta2) as well:

probably a leftover of Debian Stretch's ceph-common package?

https://packages.debian.org/stretch/amd64/ceph-common/filelist

#14 Updated by Christian Zunker almost 7 years ago

@Nathan:

I found my problem.
One of our ansible playbooks installed rados before the node had any clue about the official ceph repos. That also installed version 10.2.7-0ubuntu0.16.04.1 of ceph-common. This included the /etc/logrotate.d/ceph-common file. After adding the ceph repos the above mentioned ceph-common version was installed, but the ceph-common logrotate file not removed. This resulted in the problem of the duplicate logrotate error message.
So the problem was on my side and not with ceph or ceph-ansible.

Thanks for your time and giving me the correct hints.

#15 Updated by Nathan Cutler almost 7 years ago

  • Status changed from New to Can't reproduce

OK, thanks, closing as the Ubuntu issue was not confirmed. For the Debian issue, see #19938.

Also available in: Atom PDF