Ceph » RADOS

Right now, we keep the full pg log in memory. Each of these entries contains the name of the object involved, which means user naming patterns dramatically alter the OSD's memory usage. In #5700 we had reports of OSDs taking 1.8GB of RAM on startup with 600 PGs (which we haven't seen elsewhere), but that works out to ~1KB/log entry, which is definitely excessive.

To deal with this, we can stop storing the full pg_log_entry in memory and just keep a map or list of osd_reqid_t's. Individual log entries (or the whole thing) can be loaded on-demand when replayed ops come through or the PG has to peer.

This will reduce memory consumption and make it more predictable based on PG counts and total log sizes, at the cost of any necessary log accesses becoming more expensive (most especially, peering). Changes will need to be tested before merging for serious costs.