Ceph » mgr

The 64k limit of mon_config_key_max_entry_size is arbitrary and has previously been expanded in https://github.com/badone/ceph/commit/b38b8e980cb477ab2b0f320ab51eaa0c0fec7da6 Wondering if we just expand that again? Possibly not a dream solution. Alternatively, we could grant 'insights' module the ability to ignore that limit?

We could, but that's a big big expansion. I don't know how much to worry about monstore space consumption.

Another option is compression I suppose, perhaps at some threshold.

It's not clear at all how big this might actually get.

# ceph insights | wc
 118123  654579 7384608

The calls to 'config-key set mgr/insights' occur every 10 seconds by default and lead to a substantial entry in the log that should definitely be reviewed as well.

# grep  "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|awk '{print($1)}'|head -5
2020-11-30T06:25:02.015+0000
2020-11-30T06:25:14.159+0000
2020-11-30T06:25:26.643+0000
2020-11-30T06:25:38.231+0000
2020-11-30T06:25:52.300+0000

# grep  "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|head -1|wc
      1   12830  117254

# grep  "config-key set mgr/insights" ceph-mgr.XXX005.xxyjcw.log|tail -1|wc
      1   30097  269537

I would rather create a tiny 'insights' rados pool and dump the insights reports there. devicehealth takes this approach

We discussed this at cdm a few months ago: https://pad.ceph.com/p/insights_config-key_set_failure_problem - main conclusion was store in-memory only, and perhaps don't store every single health update