Logging section recommends fairly high levels, doesn't stress how quickly logs can fill disk
3775 introduced the observation that logs can fill very quickly and bury a small root disk.
Our documentation could perhaps
1) not recommend such high levels as it does in the example, and
2) make it some clearer just how fast logs can fill up, even without any I/O load
For instance, the example given:
[global] debug ms = 1 [mon] debug mon = 20 debug paxos = 20 debug auth = 20 [osd] debug osd = 20 debug filestore = 20 debug journal = 20 debug monc = 20 [mds] debug mds = 20 debug mds balancer = 20 debug mds log = 20 debug mds migrator = 20
mon = 20 will generate a pile of monitor messages by itself; this combination was said
to generate 1-2GB of log per hour,which is excessive. Recommend:
1) showing as an example the settings that we typically recommend for a first "what the hell is going on" type escalation past default levels
2) recommending watching log directories carefully after changing log levels to get an idea of just how much data is coming how fast
3) maybe some notes about the fact that the log level is shown on each log line, and sometimes the subsystem, so you can perhaps tune down just below messages that seem unuseful
doc: Removed fragmented logging info. Consolidated into one doc.
Logging was variously described in the ceph configuration document,
a configuration reference, and a section in operations. Since
logging and debugging are generally used with troubleshooting,
I consolidated the docs and placed them in the troubleshooting
section. Also fixed the example and provided additional detail.
Signed-off-by: John Wilkins <firstname.lastname@example.org>
#4 Updated by John Wilkins over 7 years ago
- Status changed from In Progress to Resolved
Moved the consolidated pages to http://ceph.com/docs/next/rados/troubleshooting/log-and-debug/. Will appear in master with Cuttlefish release.