Bug #48269
closed
insights module can generate too much data, fail to put in config-key
Added by Dan Mick over 3 years ago.
Updated over 2 years ago.
Backport:
nautilus,octopus,pacific
Description
A sick cluster has a lot of insights data:
- ceph insights | wc
72010 296222 3730601
The mgr logs the entire packet on "config-key set", and then shows
failed: (27) File too large
2020-11-17T06:25:09.736+0000 7f812ee58700 0 mgr set_store mon returned -27: error: entry size limited to 65536 bytes. Use 'mon config key max entry size' to manually adjust
The insights report is lost.
Not sure what to recommend for this.
- Category set to insights module
- Priority changed from Normal to Urgent
We could, but that's a big big expansion. I don't know how much to worry about monstore space consumption.
Another option is compression I suppose, perhaps at some threshold.
It's not clear at all how big this might actually get.
# ceph insights | wc
118123 654579 7384608
The calls to 'config-key set mgr/insights' occur every 10 seconds by default and lead to a substantial entry in the log that should definitely be reviewed as well.
# grep "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|awk '{print($1)}'|head -5
2020-11-30T06:25:02.015+0000
2020-11-30T06:25:14.159+0000
2020-11-30T06:25:26.643+0000
2020-11-30T06:25:38.231+0000
2020-11-30T06:25:52.300+0000
# grep "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|head -1|wc
1 12830 117254
# grep "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|tail -1|wc
1 30097 269537
I would rather create a tiny 'insights' rados pool and dump the insights reports there. devicehealth takes this approach
- Assignee set to Brad Hubbard
- Priority changed from Urgent to High
- Priority changed from High to Urgent
- Pull request ID set to 42442
- Has duplicate Bug #51637: mgr/insights: mgr consumes excessive amounts of memory added
- Status changed from New to Fix Under Review
- Target version set to v14.2.23
- Backport set to nautilus pacific
- Affected Versions v14.2.22 added
- Backport changed from nautilus pacific to nautilus,octopus,pacific
- Status changed from Fix Under Review to Pending Backport
- Copied to Backport #51949: octopus: insights module can generate too much data, fail to put in config-key added
- Copied to Backport #51950: nautilus: insights module can generate too much data, fail to put in config-key added
- Copied to Backport #51951: pacific: insights module can generate too much data, fail to put in config-key added
- Target version deleted (
v14.2.23)
- Status changed from Pending Backport to Resolved
Also available in: Atom
PDF