Project

General

Profile

Documentation #3804

Logging section recommends fairly high levels, doesn't stress how quickly logs can fill disk

Added by Dan Mick almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

3775 introduced the observation that logs can fill very quickly and bury a small root disk.
Our documentation could perhaps

1) not recommend such high levels as it does in the example, and
2) make it some clearer just how fast logs can fill up, even without any I/O load

For instance, the example given:


[global]
        debug ms = 1

[mon]
        debug mon = 20
        debug paxos = 20
        debug auth = 20

[osd]
        debug osd = 20
        debug filestore = 20
        debug journal = 20
        debug monc = 20

[mds]
        debug mds = 20
        debug mds balancer = 20
        debug mds log = 20
        debug mds migrator = 20

mon = 20 will generate a pile of monitor messages by itself; this combination was said
to generate 1-2GB of log per hour,which is excessive. Recommend:

1) showing as an example the settings that we typically recommend for a first "what the hell is going on" type escalation past default levels
2) recommending watching log directories carefully after changing log levels to get an idea of just how much data is coming how fast
3) maybe some notes about the fact that the log level is shown on each log line, and sometimes the subsystem, so you can perhaps tune down just below messages that seem unuseful


Related issues

Related to Ceph - Feature #3775: log: stop logging in statfs reports usage above some threshold New

Associated revisions

Revision 808ad25a (diff)
Added by John Wilkins over 7 years ago

doc: Removed fragmented logging info. Consolidated into one doc.

Logging was variously described in the ceph configuration document,
a configuration reference, and a section in operations. Since
logging and debugging are generally used with troubleshooting,
I consolidated the docs and placed them in the troubleshooting
section. Also fixed the example and provided additional detail.

fixes: #3804

Signed-off-by: John Wilkins <>

History

#1 Updated by Sage Weil almost 8 years ago

  • Priority changed from Normal to High

#2 Updated by John Wilkins over 7 years ago

  • Assignee set to John Wilkins

#3 Updated by John Wilkins over 7 years ago

  • Status changed from New to In Progress

Restructuring the entire logging section.

#4 Updated by John Wilkins over 7 years ago

  • Status changed from In Progress to Resolved

Moved the consolidated pages to http://ceph.com/docs/next/rados/troubleshooting/log-and-debug/. Will appear in master with Cuttlefish release.

Also available in: Atom PDF