Project

General

Profile

Bug #250

mon: delete old states to avoid filling disk

Added by Sage Weil about 10 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

We trim the pgmap (which gets the most activity), but not the others: logs, mdsmap, osdmap, etc. Unfortunately the others are trickier. mds may be safe, but for the osdmap the clients/osds currently needs a full history from where they are to present in order to behave. Need to be careful about throwing out old states. Maybe a min age would help?

History

#1 Updated by Sage Weil about 10 years ago

  • Target version set to 12

#2 Updated by Wido den Hollander about 9 years ago

I keep running into this, my monitors keep running out of inodes which causes them to stall.

Right now I have cron's like:

@daily find /var/lib/ceph/mon.logger/logm -type f -ctime +7 -delete

pgmap already gets trimmed, but the logm doesn't.

I was thinking about a cron which runs every day and ships with the Ceph packages, would that be OK? Or do you prefer to implement this in the monitor itself?

#3 Updated by Greg Farnum about 9 years ago

We should probably just do this in the monitor. The logm is never read so it's safe to delete whenever, unlike the other maps we're concerned about.

#4 Updated by Sage Weil about 9 years ago

log trimming fixed by 4c1cb28cb6cbb3d23a1736d02f6af524481c886a

osdmap, mdsmap, monmap still need trimming, but that's a whole 'nother thing.

#5 Updated by Sage Weil almost 9 years ago

  • Target version deleted (12)

#6 Updated by Sage Weil almost 9 years ago

  • Target version set to v0.38
  • translation missing: en.field_position deleted (569)
  • translation missing: en.field_position set to 958

#7 Updated by Sage Weil almost 9 years ago

  • Status changed from New to Resolved
  • translation missing: en.field_position deleted (958)
  • translation missing: en.field_position set to 958

#8 Updated by Sage Weil almost 9 years ago

  • translation missing: en.field_position deleted (958)
  • translation missing: en.field_position set to 958

Also available in: Atom PDF