Project

General

Profile

Bug #55304

libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator

Added by Ganesh Maharaj Mahalingam 6 months ago. Updated 20 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

8ce65856856059d1f702c1e2a13a048ab53ec669dbd74667bc5914069de394cc
c378729b55506b4208e12e7c1fa4fc14e459efd2f9c2c9e59518f9e3675f5b12


Description

When compiled with gcc12 (fedora36 & clearlinux) and using vstart.sh to test ceph, ceph-mgr would crash with the below error.

ceph version 17.0.0-11503-g4ba019b10bd (4ba019b10bd732774738ef26b4e32b11ca0fc3e7) quincy (dev)
1: /lib64/libc.so.6(0x42ac0) [0x7f72c5b91ac0]
2: /lib64/libc.so.6(+0x92e3c) [0x7f72c5be1e3c]
3: raise()
4: abort()
5: /lib64/libstdc
+.so.6(0xa2b77) [0x7f72c5ef1b77]
6: /lib64/libstdc
+.so.6(0xae45c) [0x7f72c5efd45c]
7: /lib64/libstdc
+.so.6(0xae4c7) [0x7f72c5efd4c7]
8: /lib64/libstdc
+.so.6(+0xae728) [0x7f72c5efd728]
9: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x48) [0x55b6edfc60ac]
10: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x52d) [0x5
5b6ee01fddd]
11: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0xc6) [0x55b6ee01ff36]
12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x45) [0x55b6ee022b05]
13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x118) [0x55b6ee023548]
14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x2d) [0x55b6ee0236bd]
15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x4d1) [0x55b6ee023041]
16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x118) [0x55b6ee023548]
17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x2d) [0x55b6ee0236bd]
22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x256) [0x55b6ee023ac6]
23: /data/workspace/ceph/ceph/build/lib/libcephsqlite.so(+0x162ea) [0x7f72c770e2ea]
24: /data/workspace/ceph/ceph/build/lib/libcephsqlite.so(+0x1a6c7) [0x7f72c77126c7]
25: /lib64/libsqlite3.so.0(+0x2e6e5) [0x7f72c75d86e5]
26: /lib64/libsqlite3.so.0(+0xadeb7) [0x7f72c7657eb7]
27: /usr/lib64/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so(+0x946c) [0x7f72bb04846c]
28: /lib64/libpython3.10.so.1.0(+0x119383) [0x7f72c7375383]
29: _PyObject_Call()
30: /usr/lib64/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so(+0x92c3) [0x7f72bb0482c3]
31: /lib64/libpython3.10.so.1.0(+0x11d101) [0x7f72c7379101]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this

backtrace on the crash yielded this.

Thread 37 "devicehealth" hit Catchpoint 1 (exception thrown), 0x00007f0f9c9b36f1 in _cxxabiv1::_cxa_throw (obj=0x5597c4783f20, tinfo=0x7f0f9cb2c360 <typeinfo for std::regex_error>, dest=0x7f0f9c9dfff0 <std::regex_error::~regex_error()>) at ../../../../libstdc++-v3/libsupc
+/eh_throw.cc:81
81 PROBE2 (throw, obj, tinfo);
(gdb) bt
#0 0x00007f0f9c9b36f1 in _cxxabiv1::_cxa_throw (obj=0x5597c4783f20, tinfo=0x7f0f9cb2c360 <typeinfo for std::regex_error>,
dest=0x7f0f9c9dfff0 <std::regex_error::~regex_error()>) at ../../../../libstdc
+-v3/libsupc++/eh_throw.cc:81
#1 0x00005597bc3b20ac in std::__throw_regex_error (_ecode=_ecode@entry=std::regex_constants::_S_error_range,
_what=_what@entry=0x5597bc6b3d30 "Invalid start of '[x-x]' range in regular expression")
at /usr/include/c++/12/bits/regex_error.h:169
#2 0x00005597bc40bddd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false> (
this=this@entry=0x7f0f775011d0, _last_char=..., __matcher=...) at /usr/include/c++/12/bits/regex_compiler.tcc:497
#3 0x00005597bc40bf36 in std::
_detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false> (
this=this@entry=0x7f0f775011d0, _neg=<optimized out>) at /usr/include/c++/12/bits/regex_compiler.tcc:419
#4 0x00005597bc40eb05 in std::
_detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression (
this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:350
#5 0x00005597bc40ebff in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom (this=this@entry=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:336
#6 0x00005597bc40f548 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_term (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:133
#7 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:115
#8 0x00005597bc40f6bd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction (
this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:91
#9 0x00005597bc40f041 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom (this=this@entry=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:329
#10 0x00005597bc40f548 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_term (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:133
#11 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:115
#12 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#13 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:112
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#15 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#16 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#17 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#18 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#19 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#20 0x00005597bc40f6bd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction (this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:91
#21 0x00005597bc40fac6 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler (this=0x7f0f775011d0, _b=<optimized out>, __e=<optimized out>, __loc=..., __flags=<optimized out>) at /usr/include/c++/12/bits/regex_compiler.tcc:76
#22 0x00007f0f9e1c42ea in std::
_cxx11::basic_regex<char, std::__cxx11::regex_traits<char> >::_M_compile (_f=std::regex_constants::_S_ECMAScript, __last=0x7f0f9e1f4118 "", __first=0x7f0f9e1f40e0 "^/*(\\*[[:digit:]]+):([[:alnum:]-.]*)/([[:alnum:]-._]+)$",
this=0x7f0f9e203e20 <parsepath(std::basic_string_view<char, std::char_traits<char> >, cephsqlite_fileloc*)::re1>) at /usr/include/c++/12/bits/regex.h:786
#23 std::__cxx11::basic_regex<char, std::__cxx11::regex_traits<char> >::basic_regex (_f=std::regex_constants::_S_ECMAScript, __p=0x7f0f9e1f40e0 "^/*(\\*[[:digit:]]+):([[:alnum:]-.]*)/([[:alnum:]-._]+)$",
this=0x7f0f9e203e20 <parsepath(std::basic_string_view<char, std::char_traits<char> >, cephsqlite_fileloc*)::re1>) at /usr/include/c++/12/bits/regex.h:453
#24 parsepath (fileloc=0x7f0f77501460, Python Exception <class 'gdb.error'>: value has been optimized out
path=) at /data/workspace/ceph/ceph/src/libcephsqlite.cc:384
#25 0x00007f0f9e1c86c7 in FullPathname (vfs=0x5597c4879ad0, ipath=0x5597c48a394c "/.mgr:devicehealth/main.db", opathlen=4097, opath=0x5597c5232008 "") at /data/workspace/ceph/ceph/src/libcephsqlite.cc:669

I created a small test.cc file with contents as below.

#include <regex>

int main() {
static const std::regex re1{"^/*(\\*[[:digit:]]+):([[:alnum:]-_.]*)/([[:alnum:]\\-._]+)$"};
}

My test results are as such.

ganeshma@ganeshma-lab2 <> temp % g++-10 -o test test.cc && ./test
ganeshma@ganeshma-lab2 <> temp % g++ -o test test.cc && ./test
terminate called after throwing an instance of 'std::regex_error'
what(): Invalid start of '[x-x]' range in regular expression
Aborted (core dumped)


Related issues

Related to RADOS - Tasks #56952: Set mgr_pool to true for a handful of tests in the rados qa suite In Progress
Duplicated by cephsqlite - Bug #55696: vstart hangs on when creating volume Duplicate
Duplicated by RADOS - Bug #56700: MGR pod on CLBO on rook deployment Duplicate
Duplicated by mgr - Bug #56745: unrecognized option ssl_server_port appears to cause failure of MGR Duplicate
Duplicated by mgr - Bug #56726: Ceph Mgr crash due to missing OSD service Duplicate
Copied to cephsqlite - Backport #56701: pacific: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator Resolved
Copied to cephsqlite - Backport #56702: quincy: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator Resolved

History

#1 Updated by Ganesh Maharaj Mahalingam 6 months ago

Please let me know if https://github.com/ceph/ceph/pull/45858 is the right fix for this issue.

#2 Updated by Patrick Donnelly 5 months ago

  • Status changed from New to Fix Under Review
  • Target version set to v18.0.0
  • Source set to Community (dev)
  • Backport set to quincy,pacific
  • Pull request ID set to 45858

#3 Updated by Patrick Donnelly 4 months ago

  • Duplicated by Bug #55696: vstart hangs on when creating volume added

#4 Updated by Neha Ojha 2 months ago

  • Status changed from Fix Under Review to Pending Backport

#5 Updated by Backport Bot 2 months ago

  • Copied to Backport #56701: pacific: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added

#6 Updated by Backport Bot 2 months ago

  • Copied to Backport #56702: quincy: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added

#7 Updated by Vikhyat Umrao 2 months ago

  • Duplicated by Bug #56700: MGR pod on CLBO on rook deployment added

#8 Updated by Laura Flores 2 months ago

I am hitting a bunch of these failures on a recent teuthology run I scheduled. The ceph version is 17.2.0:

http://pulpito.front.sepia.ceph.com/lflores-2022-07-25_21:46:45-fs:thrash:workloads-wip-telemetry-upgrade-distro-default-smithi/

2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr:terminate called after throwing an instance of 'std::regex_error'
2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr:  what():  Invalid start of range in bracket expression.
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr:*** Caught signal (Aborted) **
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr: in thread 7f1922ff6700 thread_name:devicehealth
2022-07-25T22:16:04.370 INFO:tasks.ceph.mgr.x.smithi188.stderr: ceph version 17.2.0-661-ge6650b3c (e6650b3cd2b2626a7724085062c110eb4de431b5) quincy (stable)
2022-07-25T22:16:04.371 INFO:tasks.ceph.mgr.x.smithi188.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f198c818ce0]
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 2: gsignal()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 3: abort()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f198bc1709b]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f198bc1d53c]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f198bc1d597]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f198bc1d7f8]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x55d1dc419f5a]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x55d1dc425118]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x55d1dc426726]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x55d1dc42a6ab]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x55d1dc42a78a]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x55d1dc42aa58]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x55d1dc42b43b]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 23: /lib64/libcephsqlite.so(+0x1b7fa) [0x7f198f0047fa]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 24: /lib64/libcephsqlite.so(+0x244b6) [0x7f198f00d4b6]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f198ed4af1c]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f198edb24c9]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 27: pysqlite_connection_init()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f198e8ccfc6]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 29: PyObject_Call()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 30: /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f197d6cd1f5]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f198e92f5f1]

#9 Updated by Laura Flores 2 months ago

  • Crash signature (v1) updated (diff)

#10 Updated by Laura Flores 2 months ago

  • Crash signature (v1) updated (diff)

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?var-sig_v2=80db7a543c3ba819098f773209d8fc5125b0a56677cb2276c09e91549e066c00&orgId=1

Telemetry has caught instances of this crash on 9 clusters, all version 17.2.2.

All crashes reported in Telemetry and in the teuthology occurrences I linked are happening on Centos 8 Stream. I don't see a direct correlation to gcc 12 since Centos 8 Stream runs gcc version 8.5.0.

The gibba cluster runs on Centos 8 Stream, and hasn't yet been upgraded to 17.2.2. So, we plan to upgrade the cluster soon and see if we can catch this failure.

#11 Updated by Andrew Davidoff 2 months ago

I am testing an upgrade from 17.2.1 to 17.2.2 and the first mgr upgraded crashes with what appears to be this same issue when it is promoted to active in preparation for the upgrade of the currently active 17.2.1 mgr.

[root@02 /]# cat /etc/os-release 
NAME="CentOS Stream" 
VERSION="8" 
ID="centos" 
ID_LIKE="rhel fedora" 
VERSION_ID="8" 
PLATFORM_ID="platform:el8" 
PRETTY_NAME="CentOS Stream 8" 
ANSI_COLOR="0;31" 
CPE_NAME="cpe:/o:centos:centos:8" 
HOME_URL="https://centos.org/" 
BUG_REPORT_URL="https://bugzilla.redhat.com/" 
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 

Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.850+0000 7f07659fc700  0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7f0778329fd0>, <progress.module.GhostEvent object at 0x7f0778329f98>, <progress.module.GhostEvent object at 0x7f0778329f60>[3451/76301]
module.GhostEvent object at 0x7f0778329f28>, <progress.module.GhostEvent object at 0x7f0778329ef0>, <progress.module.GhostEvent object at 0x7f0778329eb8>, <progress.module.GhostEvent object at 0x7f0778329e80>, <progress.module.GhostEvent object at 0x7f0778329e48>, <progress.module.GhostEvent object at 0x7f0778329e10>, <progr
ess.module.GhostEvent object at 0x7f0778329dd8>, <progress.module.GhostEvent object at 0x7f0778329da0>, <progress.module.GhostEvent object at 0x7f077839b208>, <progress.module.GhostEvent object at 0x7f077839b518>] historic events
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.850+0000 7f076ca0a700  0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO (2/5)
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: terminate called after throwing an instance of 'std::regex_error'
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.854+0000 7f07659fc700  0 [progress INFO root] Loaded OSDMap, ready.
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:   what():  Invalid start of range in bracket expression.
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: *** Caught signal (Aborted) **
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  in thread 7f0768201700 thread_name:devicehealth
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  2: gsignal()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  3: abort()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa
lse>&)+0x578) [0x560cee89a0a8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  27: pysqlite_connection_init()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  29: PyObject_Call()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.858+0000 7f0768201700 -1 *** Caught signal (Aborted) **
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  in thread 7f0768201700 thread_name:devicehealth
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  2: gsignal()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  3: abort()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa
lse>&)+0x578) [0x560cee89a0a8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  27: pysqlite_connection_init()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  29: PyObject_Call()
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1]
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: --- begin dump of recent events ---
...
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: debug      0> 2022-07-27T16:07:27.858+0000 7f0768201700 -1 *** Caught signal (Aborted) **                                                                                                                                                                      
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  in thread 7f0768201700 thread_name:devicehealth                                                                                                                                                                                                               
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)                                                                                                                                                                                
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0]                                                                                                                                                                                                          
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  2: gsignal()                                                                                                                                                                                                                                                  
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  3: abort()                                                                                                                                                                                                                                                    
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b]                                                                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c]                                                                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597]                                                                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8]                                                                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea]                                                                                                                                                            
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa
lse>&)+0x578) [0x560cee89a0a8]                                                                                                                                                                                                                                                                                                        
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6]                                                                                                                  
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b]                                                                                                                                              
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a]                                                                                                                                                            
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8]                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0]                                                                                                                                                     
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb]                                                                   
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca]                                                                                                                                                                                                        
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486]                                                                                                                                                                                                        
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c]                                                                                                                                                                                                         
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9]                                                                                                                                                                                                         
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  27: pysqlite_connection_init()                                                                                                                                                                                                                                
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6]                                                                                                                                                                                                   
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  29: PyObject_Call()                                                                                                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5]                                                                                                                                                           
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1]                                                                                                                                                                                                   
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.                                                                                                                                                                   
Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: --- logging levels ---

#12 Updated by Laura Flores 2 months ago

Hi @Andrew,

There is a thread about this issue on the ceph-users mailing list (subject is "[ceph-users] 17.2.2: all MGRs crashing in fresh cephadm install"). Here is some info about the issue in case you are interested in testing a fix:

This issue seems to be showing up in 17.2.2, details in
https://tracker.ceph.com/issues/55304. We are currently in the process
of validating the fix https://github.com/ceph/ceph/pull/47270 and
we'll try to expedite a quick fix.

In the meantime, we have builds/images of the dev version of the fix,
in case you want to give it a try.
https://shaman.ceph.com/builds/ceph/wip-quincy-libcephsqlite-fix/
quay.ceph.io/ceph-ci/ceph:f516549e3e4815795ff0343ab71b3ebf567e5531

#14 Updated by Telemetry Bot 2 months ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v17.2.2 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=80db7a543c3ba819098f773209d8fc5125b0a56677cb2276c09e91549e066c00

Sanitized backtrace:

    std::__throw_regex_error(std::regex_constants::error_type, char const*)
    bool std::__detail::_Compiler<std::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::regex_traits<char>, false, false>&)
    void std::__detail::_Compiler<std::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)
    pysqlite_connection_init()
    PyObject_Call()

Crash dump sample:
{
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7f95c2b3bce0]",
        "gsignal()",
        "abort()",
        "/lib64/libstdc++.so.6(+0x9009b) [0x7f95c1f3a09b]",
        "/lib64/libstdc++.so.6(+0x9653c) [0x7f95c1f4053c]",
        "/lib64/libstdc++.so.6(+0x96597) [0x7f95c1f40597]",
        "/lib64/libstdc++.so.6(+0x967f8) [0x7f95c1f407f8]",
        "(std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x5641b965deea]",
        "(bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x5641b96690a8]",
        "(void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x5641b966a6b6]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x5641b966e63b]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x5641b966e71a]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x5641b966eca0]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x5641b966edf0]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x5641b966e9e8]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x5641b966eca0]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x5641b966edf0]",
        "(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x5641b966f3cb]",
        "/lib64/libcephsqlite.so(+0x1b7ca) [0x7f95c53257ca]",
        "/lib64/libcephsqlite.so(+0x24486) [0x7f95c532e486]",
        "/lib64/libsqlite3.so.0(+0x75f1c) [0x7f95c506bf1c]",
        "/lib64/libsqlite3.so.0(+0xdd4c9) [0x7f95c50d34c9]",
        "pysqlite_connection_init()",
        "/lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f95c4bedfc6]",
        "PyObject_Call()",
        "/usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f95b385e1f5]",
        "/lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f95c4c505f1]" 
    ],
    "ceph_version": "17.2.2",
    "crash_id": "2022-07-25T23:51:19.181675Z_55d26f06-f310-4036-9b47-6019814ac322",
    "entity_name": "mgr.d3c45872f563d8b9ab59b23d8f89dbfe9fc3806e",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-mgr",
    "stack_sig": "c378729b55506b4208e12e7c1fa4fc14e459efd2f9c2c9e59518f9e3675f5b12",
    "timestamp": "2022-07-25T23:51:19.181675Z",
    "utsname_machine": "x86_64",
    "utsname_release": "3.10.0-1160.36.2.el7.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Wed Jul 21 11:57:15 UTC 2021" 
}

#15 Updated by Laura Flores 2 months ago

  • Crash signature (v1) updated (diff)

We ran a teuthology reproducer on 17.2.1 and 17.2.2 respectively:

17.2.1
https://shaman.ceph.com/builds/ceph/quincy/ec95624474b1871a821a912b8c3af68f8f8e7aa1/
http://pulpito.front.sepia.ceph.com/lflores-2022-07-28_14:26:31-fs:thrash:workloads-quincy-distro-default-smithi/

17.2.2
https://shaman.ceph.com/builds/ceph/quincy/b6e46b8939c67a6cc754abb4d0ece3c8918eccc3/
http://pulpito.front.sepia.ceph.com/lflores-2022-07-28_14:29:53-fs:thrash:workloads-quincy-distro-default-smithi/

From the 17.2.1 build:

Processing files: libcephsqlite-17.2.1-0.el8.x86_64
Provides: libcephsqlite = 2:17.2.1-0.el8 libcephsqlite(x86-64) = 2:17.2.1-0.el8 libcephsqlite.so()(64bit)

From the 17.2.2 build:

Processing files: libcephsqlite-17.2.2-0.el8.x86_64
warning: Ignoring invalid regex ^%{_scl_prefix}/.*|%{_root_sysconfdir}/rpm/macros.gcc-toolset-11-config$
warning: Ignoring invalid regex ^%{_root_sysconfdir}/rpm/macros.gcc-toolset-11-config$
Provides: libcephsqlite = 2:17.2.2-0.el8 libcephsqlite(x86-64) = 2:17.2.2-0.el8 libcephsqlite.so()(64bit)

We found the issue to be that 17.2.1 was built using gcc 8.5.0-13, while 17.2.2 used 8.5.0-14.
https://bugzilla.redhat.com/show_bug.cgi?id=2001788 is the only difference.
See the changelong here: https://centos.pkgs.org/8-stream/centos-appstream-x86_64/gcc-8.5.0-14.el8.x86_64.rpm.html

This bug was not caught on the RC because the RC build likely got built using gcc 8.5.0-13. The build is gone now though, so there is no way to tell for sure.

#16 Updated by Laura Flores 2 months ago

  • Related to Tasks #56952: Set mgr_pool to true for a handful of tests in the rados qa suite added

#18 Updated by Brian Woods 2 months ago

I may also be seeing this issue, but on a completely fresh deployment somehow.

https://tracker.ceph.com/issues/56745?next_issue_id=56971

Going to zero everything again and do one more clean deployment just to be 100% sure.

#19 Updated by Laura Flores 2 months ago

Hi @Brian, looks like you are indeed hitting this same issue. This crash occurs on 17.2.2 clusters due to a change in gcc version (at build time). There will be a hotfix release available soon to address the issue. See https://ceph.io/en/news/blog/2022/v17-2-3-quincy-released/ for more details.

#20 Updated by Brian Woods 2 months ago

@Laura, all good! Seems I had some crazy timing of some sort (maybe the cephadm on GIT hasn't been updated???). I had intended to deploy 17.2.3 from the beginning, but somehow got 17.2.2 but then deployed 17.2.3.... IDK How really. SO I am all good, and if that is the case, I am going to cancel my ticket as a duplicate.

#21 Updated by Konstantin Shalygin 2 months ago

  • Duplicated by Bug #56745: unrecognized option ssl_server_port appears to cause failure of MGR added

#23 Updated by Andrew Davidoff about 2 months ago

Thanks @Laura! Apologies that I didn't see your updates until just now, otherwise I would have tested as you suggested.

#24 Updated by Backport Bot about 2 months ago

  • Tags set to backport_processed

#25 Updated by Radoslaw Zarzynski 21 days ago

  • Duplicated by Bug #56726: Ceph Mgr crash due to missing OSD service added

#26 Updated by Konstantin Shalygin 20 days ago

  • Status changed from Pending Backport to Resolved

#27 Updated by Konstantin Shalygin 20 days ago

  • Tags deleted (backport_processed)

Also available in: Atom PDF