Ceph » devops

I hit this issue under a EL7 build since it pulled in an old version:

Jenkins log:

...
 --> Already installed : devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64
...

GDB:

(gdb) bt
#0  0x00007fffeb52c7b0 in abort () from /lib64/libc.so.6
#1  0x00007fffeb56dcc7 in __libc_message () from /lib64/libc.so.6
#2  0x00007fffeb576429 in _int_free () from /lib64/libc.so.6
#3  0x00007ffff7b36b75 in std::string::_Rep::_M_dispose (__a=..., this=<optimized out>) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/stl_deque.h:1515
#4  std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x7fffffffcdd0, __in_chrg=<optimized out>) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.h:3621
#5  librados::RadosClient::connect (this=0x5555558e0940) at /usr/src/debug/ceph-14.0.1-3212-g571a8f2/src/librados/RadosClient.cc:256
#6  0x00005555555785bc in rados_tool_common (opts=std::map with 1 element = {...}, nargs=std::vector of length 1, capacity 3 = {...}) at /usr/src/debug/ceph-14.0.1-3212-g571a8f2/src/tools/rados/rados.cc:2123
#7  0x00005555555722bf in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/ceph-14.0.1-3212-g571a8f2/src/tools/rados/rados.cc:4029
(gdb) frame 4
#4  std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x7fffffffcdd0, __in_chrg=<optimized out>) at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/basic_string.h:3621
3621          { _M_rep()->_M_dispose(this->get_allocator()); }
(gdb) print /r *this
$2 = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, 
    _M_p = 0x5555557da0b8 <std::string::_Rep::_S_empty_rep_storage+24> ""}}

You are not authorized to access bug #1616223. To see this bug, you must first log in to an account with the appropriate permissions. 

Jason, does the bug report specify which minimum version is needed? The spec file currently says:

%if 0%{?rhel} == 7
# devtoolset offers newer make and valgrind-devel, but the old ones are good
# enough.
BuildRequires:  devtoolset-7-gcc-c++ >= 7.3.1
%else
BuildRequires:  gcc-c++
%endif

. . . which explains why devtoolset-7-gcc-c++-7.3.1-5.10.el7.x86_64 was installed (or, more precisely, not upgraded) in your RHEL 7/CentOS 7 environment.

Nathan Cutler wrote:

[...]

Jason, does the bug report specify which minimum version is needed? The spec file currently says:

Yes, the BZ lists that the bug is fixed in "devtoolset-7-gcc-7.3.1-5.13". (Sorry, didn't realize that it was a private BZ for whatever reason).

I used this change to fix the issue and the crashes stopped:

https://github.com/ceph/ceph/pull/26237/commits/270c04776cd42ecd1dcfad265d3aae753bbee1d3