Ceph » RADOS

While I was looking into this I noticed this warning in the Jenkins output.

JENKINS-19022: warning: possible memory leak due to Git plugin usage; see: https://wiki.jenkins-ci.org/display/JENKINS/Remove+Git+Plugin+BuildsByBranch+BuildData

I wonder whether this is a contributing factor to some of the issues we've seen?

This specific issue appears to be transient as telling Jenkins to try again appears to resolve it in most cases. This may indicate some form of resource contention as the cause.

per Brad

If we see this again we could try temporarily adding "--param ggc-min-expand=1 --param ggc-min-heapsize=0" to the compiler command line as that seems to prevent the segfault, at least in my testing.

• ggc-min-expand is the percentage of the gc'heap expansion before reaching which, gc will perform collection. its default value is @std::min(30% + 70% * (RAM/1GB), 100%). changing it to 1 will force gc to try to do gc very often -- it will try to do gc every time its heap size increases by 1%.
• ggc-min-heapsize is the absolute size of gc before it tries to perform collection. its default value is std::clamp(4MiB, RAM/8, 128MiB), where RAM is std::min_element(prlimit --as --data --rss). 0 will force GCC's gc to collect whenever possible no matter how small the heap is.

See https://github.com/ceph/ceph/pull/27479 for a viable workaround. Note that this is a bug in gcc7 [1] and the preferred fix is to move to gcc8 however I wanted to add that PR so people had access to the workaround for this, and any future occurrences.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1531154