Bug #5175
closed
leveldb: LOG and MANIFEST file grow without bound (LOG being _text_ log !)
Added by Sylvain Munaut almost 11 years ago.
Updated over 10 years ago.
Description
leveldb has two files that seem to grow without bound and are only cleared on db open.
The first is the LOG file which is a textual debug log that leveldb creates. It's opened when the db is opened and is never closed or cycled (and leveldb doesn't have any method to make it cycle other than close/reopen).
To make matter worse, if this file is on XFS, it's subject to XFS pre-allocation (so each time the file size reaches the currently on-disk allocated size, the allocated size doubles, and so you get used disk space of 2M/4M/.../256M/512M giving big jumps in used disk space as reported by 'du -sh')
The second file is the MANIFEST file which grows a little at each compaction and is also only trimmed on db open.
For long running (i.e. weeks/month), those can actually grow quite a bit (especially LOG).
Note that the same issue exists on OSD as well but they seem to receive a whole lot less updates than mon so it shows less.
Work around for LOG is to use this config :
mon leveldb log = /dev/null
- Status changed from New to 12
- Priority changed from Normal to High
i wonder if turning off the compaction will make this grow slowly enough to not be an issue. strangely, i still get a LOG file (ableit smaller) when i set it to /dev/null. :/
- Assignee set to Joao Eduardo Luis
- Status changed from 12 to 4
Here's how leveldb works in respect to the log:
- setting it to /dev/null will only make the info messages to be written to /dev/null -- leveldb will still open /dev/null on every run and will still spend time formatting debug messages to be written to the void of nothingness.
- not setting a log file will make leveldb default to <leveldb-location>/LOG.
- regardless of the location we set, leveldb will still write some stuff to LOG.
I have a logger class (to be used by leveldb) implementation (to be used by us) supporting log rotation, but after a couple of iterations it still feels wrong. We should try to get something done on leveldb's side (also started working on a patch for leveldb but am not yet familiar enough with the code to make sure everything was okay).
Anyway, I guess the simplest solution for this on the short-term is to create a logger that either logs to a file or just drops any leveldb messages in the cold. Keeping it on our side still feels a bit wrong, as one would expect to be able to turn off logging on leveldb if one so desired, but that seems impossible and feels a lot like a missing feature on leveldb itself.
Thoughts?
- Status changed from 4 to Resolved
Also available in: Atom
PDF