Project

General

Profile

Actions

Documentation #3804

closed

Logging section recommends fairly high levels, doesn't stress how quickly logs can fill disk

Added by Dan Mick over 11 years ago. Updated about 11 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

3775 introduced the observation that logs can fill very quickly and bury a small root disk.
Our documentation could perhaps

1) not recommend such high levels as it does in the example, and
2) make it some clearer just how fast logs can fill up, even without any I/O load

For instance, the example given:


[global]
        debug ms = 1

[mon]
        debug mon = 20
        debug paxos = 20
        debug auth = 20

[osd]
        debug osd = 20
        debug filestore = 20
        debug journal = 20
        debug monc = 20

[mds]
        debug mds = 20
        debug mds balancer = 20
        debug mds log = 20
        debug mds migrator = 20

mon = 20 will generate a pile of monitor messages by itself; this combination was said
to generate 1-2GB of log per hour,which is excessive. Recommend:

1) showing as an example the settings that we typically recommend for a first "what the hell is going on" type escalation past default levels
2) recommending watching log directories carefully after changing log levels to get an idea of just how much data is coming how fast
3) maybe some notes about the fact that the log level is shown on each log line, and sometimes the subsystem, so you can perhaps tune down just below messages that seem unuseful


Related issues 1 (1 open0 closed)

Related to Ceph - Feature #3775: log: stop logging in statfs reports usage above some thresholdNew

Actions
Actions #1

Updated by Sage Weil over 11 years ago

  • Priority changed from Normal to High
Actions #2

Updated by John Wilkins about 11 years ago

  • Assignee set to John Wilkins
Actions #3

Updated by John Wilkins about 11 years ago

  • Status changed from New to In Progress

Restructuring the entire logging section.

Actions #4

Updated by John Wilkins about 11 years ago

  • Status changed from In Progress to Resolved

Moved the consolidated pages to http://ceph.com/docs/next/rados/troubleshooting/log-and-debug/. Will appear in master with Cuttlefish release.

Actions

Also available in: Atom PDF