Bug #38892: /ceph/src/tools/kvstore_tool.cc:266:1: internal compiler error: Segmentation fault - RADOS - Ceph

Actions

Copy link

Bug #38892

closed

/ceph/src/tools/kvstore_tool.cc:266:1: internal compiler error: Segmentation fault

Added by Sebastian Wagner about 5 years ago. Updated over 4 years ago.

Status:

Closed

Priority:

Low

Assignee:

Brad Hubbard

Category:

Code Hygiene

Target version:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

26828

Crash signature (v1):

Crash signature (v2):

Description

[ 79%] Building CXX object src/tools/CMakeFiles/ceph-kvstore-tool.dir/kvstore_tool.cc.o
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/kvstore_tool.cc:266:1: internal compiler error: Segmentation fault
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/cc51Ovir.out file, please attach this to your bugreport.

This is line 266 this time.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Copied from Bug #38633: /ceph/src/tools/kvstore_tool.cc:39:1: internal compiler error: Segmentation fault on Bionic added

Actions

Copy link

Updated by Sebastian Wagner about 5 years ago

Description updated (diff)

Actions

Copy link

Updated by Brad Hubbard about 5 years ago

While I was looking into this I noticed this warning in the Jenkins output.

JENKINS-19022: warning: possible memory leak due to Git plugin usage; see: https://wiki.jenkins-ci.org/display/JENKINS/Remove+Git+Plugin+BuildsByBranch+BuildData

I wonder whether this is a contributing factor to some of the issues we've seen?

This specific issue appears to be transient as telling Jenkins to try again appears to resolve it in most cases. This may indicate some form of resource contention as the cause.

Actions

Copy link

Updated by Kefu Chai about 5 years ago

per Brad

If we see this again we could try temporarily adding "--param ggc-min-expand=1 --param ggc-min-heapsize=0" to the compiler command line as that seems to prevent the segfault, at least in my testing.

ggc-min-expand is the percentage of the gc'heap expansion before reaching which, gc will perform collection. its default value is @std::min(30% + 70% * (RAM/1GB), 100%). changing it to 1 will force gc to try to do gc very often -- it will try to do gc every time its heap size increases by 1%.
ggc-min-heapsize is the absolute size of gc before it tries to perform collection. its default value is std::clamp(4MiB, RAM/8, 128MiB), where RAM is std::min_element(prlimit --as --data --rss). 0 will force GCC's gc to collect whenever possible no matter how small the heap is.

Actions

Copy link

Updated by Brad Hubbard about 5 years ago

See https://github.com/ceph/ceph/pull/27479 for a viable workaround. Note that this is a bug in gcc7 [1] and the preferred fix is to move to gcc8 however I wanted to add that PR so people had access to the workaround for this, and any future occurrences.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1531154

Actions

Copy link