/ceph/src/tools/kvstore_tool.cc:266:1: internal compiler error: Segmentation fault
[ 79%] Building CXX object src/tools/CMakeFiles/ceph-kvstore-tool.dir/kvstore_tool.cc.o /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/kvstore_tool.cc:266:1: internal compiler error: Segmentation fault } ^ Please submit a full bug report, with preprocessed source if appropriate. See <http://bugzilla.redhat.com/bugzilla> for instructions. Preprocessed source stored into /tmp/cc51Ovir.out file, please attach this to your bugreport.
This is line 266 this time.
#3 Updated by Brad Hubbard over 3 years ago
While I was looking into this I noticed this warning in the Jenkins output.
JENKINS-19022: warning: possible memory leak due to Git plugin usage; see: https://wiki.jenkins-ci.org/display/JENKINS/Remove+Git+Plugin+BuildsByBranch+BuildData
I wonder whether this is a contributing factor to some of the issues we've seen?
This specific issue appears to be transient as telling Jenkins to try again appears to resolve it in most cases. This may indicate some form of resource contention as the cause.
#4 Updated by Kefu Chai over 3 years ago
If we see this again we could try temporarily adding "--param ggc-min-expand=1 --param ggc-min-heapsize=0" to the compiler command line as that seems to prevent the segfault, at least in my testing.
- ggc-min-expand is the percentage of the gc'heap expansion before reaching which, gc will perform collection. its default value is @std::min(30% + 70% * (RAM/1GB), 100%). changing it to 1 will force gc to try to do gc very often -- it will try to do gc every time its heap size increases by 1%.
- ggc-min-heapsize is the absolute size of gc before it tries to perform collection. its default value is
std::clamp(4MiB, RAM/8, 128MiB), where RAM is
std::min_element(prlimit --as --data --rss). 0 will force GCC's gc to collect whenever possible no matter how small the heap is.
#5 Updated by Brad Hubbard over 3 years ago
See https://github.com/ceph/ceph/pull/27479 for a viable workaround. Note that this is a bug in gcc7  and the preferred fix is to move to gcc8 however I wanted to add that PR so people had access to the workaround for this, and any future occurrences.