Project

General

Profile

Bug #12416

The cthulhu should have the thead clear the database and log.

Added by ceph zte over 8 years ago. Updated over 8 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Now the calamari always write data to the postgresql and write log to /var/log/calamari

Maybe after a long time ,there are a lot of no use data in postgressql and a lot of no use log in /var/log/calamari.

The cthulhu should have a thread to delete the no use data in postgresql and no use log in /var/log/calamari.

It can be configed in the /etc/calamari/cthulhu.conf how long data or log we keep.

History

#1 Updated by Dan Mick over 8 years ago

Log file rotation should be set up and configured with logrotate. I don't think the postgres data grows without bound. Do you have any evidence that it's growing?

#2 Updated by ceph zte over 8 years ago

The cthulhu_event in my machine is now have so many data.

And the cthulhu_event table will keep on growing.

I think we should have a thead to clear the table , maybe some day the table is full.

calamari=# select count(*) from cthulhu_event;
----- | count |
----- | 37059 |
-----
1 row in set

5 Calamari server started
5 Added server song with 3 OSDs, 1 monitor service bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Server song is late reporting in, last report at 2015-07-03 20:10:30.719684+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
4 Server song regained contact bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:12:39.820499+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
5 Calamari server started
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
5 Added server song with 3 OSDs, 1 monitor service bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:16:32.149754+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
4 Server song regained contact bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:18:40.387047+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
4 Server song regained contact bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:20:51.821198+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
4 Server song regained contact bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:23:01.438075+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
4 Server song regained contact bc253614-d8aa-48ab-8b75-996e152d1222 song
4 Cluster 'ceph' regained contact bc253614-d8aa-48ab-8b75-996e152d1222
3 Server song is late reporting in, last report at 2015-07-03 20:25:13.126374+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
5 Calamari server started
5 Added server song with 3 OSDs, 1 monitor service bc253614-d8aa-48ab-8b75-996e152d1222 song
3 Server song is late reporting in, last report at 2015-07-03 20:55:35.872227+00:00 bc253614-d8aa-48ab-8b75-996e152d1222 song
5 Calamari server started
3 Cluster 'ceph' is late reporting in bc253614-d8aa-48ab-8b75-996e152d1222
5 Added server song with 3 OSDs, 1 monitor service bc253614-d8aa-48ab-8b75-996e152d1222 song

#3 Updated by Christina Meno over 8 years ago

Thanks for the report!

#4 Updated by ceph zte over 8 years ago

I have write the clear log and evevt table code.

class ClearlHistory(gevent.greenlet.Greenlet):
def init(self):
super(ClearlHistory, self).__init__()
self._complete = gevent.event.Event()

def stop(self):
self._complete.set()
def clearlog(self):
logfile = os.listdir("/var/log/calamari/")
from datetime import date
for elem in logfile:
m = re.search(r"gz", elem)
if m:
history = elem[-11:-3]
histtime = date(int(history[0:4]), int(history[4:6]), int(history[6:8]))
diff = (date.today() - histtime).days
if int(diff) > 14:
file = "rm /var/log/calamari/" + elem
os.system(file)
def cleardatabase(self):
if sqlalchemy is not None:
try: # Prepare persistence
engine = create_engine(config.get('cthulhu', 'db_path'))
Session.configure(bind=engine)
except sqlalchemy.exc.ArgumentError as e:
log.error("Database error: %s" % e)
raise
try:
from calamari_common.db.event import Event
from dateutil import tz
import datetime
session = Session()
now = datetime.datetime.utcnow().replace(tzinfo=tz.tzutc())
now = now + datetime.timedelta(days=-90)
session.query(Event).filter(Event.when<now).delete()
object_types = [row[0] for row in session.query(SyncObject.sync_type).distinct()]
for sync in object_types:
session.query(SyncObject).filter_by(sync_type=sync).filter(SyncObject.when<now).delete()
session.commit()
except:
session.rollback()
def Run(self):
self.clearlog()
self.cleardatabase()

Also available in: Atom PDF