Bug #55304
closedlibcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator
0%
8ce65856856059d1f702c1e2a13a048ab53ec669dbd74667bc5914069de394cc
c378729b55506b4208e12e7c1fa4fc14e459efd2f9c2c9e59518f9e3675f5b12
Description
When compiled with gcc12 (fedora36 & clearlinux) and using vstart.sh to test ceph, ceph-mgr would crash with the below error.
ceph version 17.0.0-11503-g4ba019b10bd (4ba019b10bd732774738ef26b4e32b11ca0fc3e7) quincy (dev)
1: /lib64/libc.so.6(0x42ac0) [0x7f72c5b91ac0]
2: /lib64/libc.so.6(+0x92e3c) [0x7f72c5be1e3c]
3: raise()
4: abort()
5: /lib64/libstdc+.so.6(0xa2b77) [0x7f72c5ef1b77]
6: /lib64/libstdc+.so.6(0xae45c) [0x7f72c5efd45c]
7: /lib64/libstdc+.so.6(0xae4c7) [0x7f72c5efd4c7]
8: /lib64/libstdc+.so.6(+0xae728) [0x7f72c5efd728]
9: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x48) [0x55b6edfc60ac]
10: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x52d) [0x5
5b6ee01fddd]
11: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0xc6) [0x55b6ee01ff36]
12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x45) [0x55b6ee022b05]
13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x118) [0x55b6ee023548]
14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x2d) [0x55b6ee0236bd]
15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x4d1) [0x55b6ee023041]
16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x118) [0x55b6ee023548]
17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x71) [0x55b6ee0234a1]
21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x2d) [0x55b6ee0236bd]
22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x256) [0x55b6ee023ac6]
23: /data/workspace/ceph/ceph/build/lib/libcephsqlite.so(+0x162ea) [0x7f72c770e2ea]
24: /data/workspace/ceph/ceph/build/lib/libcephsqlite.so(+0x1a6c7) [0x7f72c77126c7]
25: /lib64/libsqlite3.so.0(+0x2e6e5) [0x7f72c75d86e5]
26: /lib64/libsqlite3.so.0(+0xadeb7) [0x7f72c7657eb7]
27: /usr/lib64/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so(+0x946c) [0x7f72bb04846c]
28: /lib64/libpython3.10.so.1.0(+0x119383) [0x7f72c7375383]
29: _PyObject_Call()
30: /usr/lib64/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so(+0x92c3) [0x7f72bb0482c3]
31: /lib64/libpython3.10.so.1.0(+0x11d101) [0x7f72c7379101]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this
backtrace on the crash yielded this.
Thread 37 "devicehealth" hit Catchpoint 1 (exception thrown), 0x00007f0f9c9b36f1 in _cxxabiv1::_cxa_throw (obj=0x5597c4783f20, tinfo=0x7f0f9cb2c360 <typeinfo for std::regex_error>, dest=0x7f0f9c9dfff0 <std::regex_error::~regex_error()>) at ../../../../libstdc++-v3/libsupc
+/eh_throw.cc:81
81 PROBE2 (throw, obj, tinfo);
(gdb) bt
#0 0x00007f0f9c9b36f1 in _cxxabiv1::_cxa_throw (obj=0x5597c4783f20, tinfo=0x7f0f9cb2c360 <typeinfo for std::regex_error>,
dest=0x7f0f9c9dfff0 <std::regex_error::~regex_error()>) at ../../../../libstdc+-v3/libsupc++/eh_throw.cc:81
#1 0x00005597bc3b20ac in std::__throw_regex_error (_ecode=_ecode@entry=std::regex_constants::_S_error_range,
_what=_what@entry=0x5597bc6b3d30 "Invalid start of '[x-x]' range in regular expression")
at /usr/include/c++/12/bits/regex_error.h:169
#2 0x00005597bc40bddd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false> (
this=this@entry=0x7f0f775011d0, _last_char=..., __matcher=...) at /usr/include/c++/12/bits/regex_compiler.tcc:497
#3 0x00005597bc40bf36 in std::_detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false> (
this=this@entry=0x7f0f775011d0, _neg=<optimized out>) at /usr/include/c++/12/bits/regex_compiler.tcc:419
#4 0x00005597bc40eb05 in std::_detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression (
this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:350
#5 0x00005597bc40ebff in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom (this=this@entry=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:336
#6 0x00005597bc40f548 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_term (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:133
#7 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:115
#8 0x00005597bc40f6bd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction (
this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:91
#9 0x00005597bc40f041 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom (this=this@entry=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:329
#10 0x00005597bc40f548 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_term (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:133
#11 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:115
#12 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#13 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0)
at /usr/include/c++/12/bits/regex_compiler.tcc:112
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#15 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#16 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#17 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#18 0x00005597bc40f4a1 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:118
#19 std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative (this=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:112
#20 0x00005597bc40f6bd in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction (this=this@entry=0x7f0f775011d0) at /usr/include/c++/12/bits/regex_compiler.tcc:91
#21 0x00005597bc40fac6 in std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler (this=0x7f0f775011d0, _b=<optimized out>, __e=<optimized out>, __loc=..., __flags=<optimized out>) at /usr/include/c++/12/bits/regex_compiler.tcc:76
#22 0x00007f0f9e1c42ea in std::_cxx11::basic_regex<char, std::__cxx11::regex_traits<char> >::_M_compile (_f=std::regex_constants::_S_ECMAScript, __last=0x7f0f9e1f4118 "", __first=0x7f0f9e1f40e0 "^/*(\\*[[:digit:]]+):([[:alnum:]-.]*)/([[:alnum:]-._]+)$",
this=0x7f0f9e203e20 <parsepath(std::basic_string_view<char, std::char_traits<char> >, cephsqlite_fileloc*)::re1>) at /usr/include/c++/12/bits/regex.h:786
#23 std::__cxx11::basic_regex<char, std::__cxx11::regex_traits<char> >::basic_regex (_f=std::regex_constants::_S_ECMAScript, __p=0x7f0f9e1f40e0 "^/*(\\*[[:digit:]]+):([[:alnum:]-.]*)/([[:alnum:]-._]+)$",
this=0x7f0f9e203e20 <parsepath(std::basic_string_view<char, std::char_traits<char> >, cephsqlite_fileloc*)::re1>) at /usr/include/c++/12/bits/regex.h:453
#24 parsepath (fileloc=0x7f0f77501460, Python Exception <class 'gdb.error'>: value has been optimized out
path=) at /data/workspace/ceph/ceph/src/libcephsqlite.cc:384
#25 0x00007f0f9e1c86c7 in FullPathname (vfs=0x5597c4879ad0, ipath=0x5597c48a394c "/.mgr:devicehealth/main.db", opathlen=4097, opath=0x5597c5232008 "") at /data/workspace/ceph/ceph/src/libcephsqlite.cc:669
I created a small test.cc file with contents as below.
#include <regex>
int main()
{
static const std::regex re1{"^/*(\\*[[:digit:]]+):([[:alnum:]-_.]*)/([[:alnum:]\\-._]+)$"};
}
My test results are as such.
ganeshma@ganeshma-lab2 <> temp % g++-10 -o test test.cc && ./test
ganeshma@ganeshma-lab2 <> temp % g++ -o test test.cc && ./test
terminate called after throwing an instance of 'std::regex_error'
what(): Invalid start of '[x-x]' range in regular expression
Aborted (core dumped)
Updated by Ganesh Maharaj Mahalingam about 2 years ago
Please let me know if https://github.com/ceph/ceph/pull/45858 is the right fix for this issue.
Updated by Patrick Donnelly almost 2 years ago
- Status changed from New to Fix Under Review
- Target version set to v18.0.0
- Source set to Community (dev)
- Backport set to quincy,pacific
- Pull request ID set to 45858
Updated by Patrick Donnelly almost 2 years ago
- Has duplicate Bug #55696: vstart hangs on when creating volume added
Updated by Neha Ojha over 1 year ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 1 year ago
- Copied to Backport #56701: pacific: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added
Updated by Backport Bot over 1 year ago
- Copied to Backport #56702: quincy: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added
Updated by Vikhyat Umrao over 1 year ago
- Has duplicate Bug #56700: MGR pod on CLBO on rook deployment added
Updated by Laura Flores over 1 year ago
I am hitting a bunch of these failures on a recent teuthology run I scheduled. The ceph version is 17.2.0:
2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr:terminate called after throwing an instance of 'std::regex_error'
2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr: what(): Invalid start of range in bracket expression.
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr:*** Caught signal (Aborted) **
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr: in thread 7f1922ff6700 thread_name:devicehealth
2022-07-25T22:16:04.370 INFO:tasks.ceph.mgr.x.smithi188.stderr: ceph version 17.2.0-661-ge6650b3c (e6650b3cd2b2626a7724085062c110eb4de431b5) quincy (stable)
2022-07-25T22:16:04.371 INFO:tasks.ceph.mgr.x.smithi188.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f198c818ce0]
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 2: gsignal()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 3: abort()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f198bc1709b]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f198bc1d53c]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f198bc1d597]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f198bc1d7f8]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x55d1dc419f5a]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x55d1dc425118]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x55d1dc426726]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x55d1dc42a6ab]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x55d1dc42a78a]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x55d1dc42aa58]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x55d1dc42b43b]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 23: /lib64/libcephsqlite.so(+0x1b7fa) [0x7f198f0047fa]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 24: /lib64/libcephsqlite.so(+0x244b6) [0x7f198f00d4b6]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f198ed4af1c]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f198edb24c9]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 27: pysqlite_connection_init()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f198e8ccfc6]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 29: PyObject_Call()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 30: /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f197d6cd1f5]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f198e92f5f1]
Updated by Laura Flores over 1 year ago
- Crash signature (v1) updated (diff)
Telemetry has caught instances of this crash on 9 clusters, all version 17.2.2.
All crashes reported in Telemetry and in the teuthology occurrences I linked are happening on Centos 8 Stream. I don't see a direct correlation to gcc 12 since Centos 8 Stream runs gcc version 8.5.0.
The gibba cluster runs on Centos 8 Stream, and hasn't yet been upgraded to 17.2.2. So, we plan to upgrade the cluster soon and see if we can catch this failure.
Updated by Andrew Davidoff over 1 year ago
I am testing an upgrade from 17.2.1 to 17.2.2 and the first mgr upgraded crashes with what appears to be this same issue when it is promoted to active in preparation for the upgrade of the currently active 17.2.1 mgr.
[root@02 /]# cat /etc/os-release NAME="CentOS Stream" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Stream 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.850+0000 7f07659fc700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7f0778329fd0>, <progress.module.GhostEvent object at 0x7f0778329f98>, <progress.module.GhostEvent object at 0x7f0778329f60>[3451/76301] module.GhostEvent object at 0x7f0778329f28>, <progress.module.GhostEvent object at 0x7f0778329ef0>, <progress.module.GhostEvent object at 0x7f0778329eb8>, <progress.module.GhostEvent object at 0x7f0778329e80>, <progress.module.GhostEvent object at 0x7f0778329e48>, <progress.module.GhostEvent object at 0x7f0778329e10>, <progr ess.module.GhostEvent object at 0x7f0778329dd8>, <progress.module.GhostEvent object at 0x7f0778329da0>, <progress.module.GhostEvent object at 0x7f077839b208>, <progress.module.GhostEvent object at 0x7f077839b518>] historic events Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.850+0000 7f076ca0a700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO (2/5) Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: terminate called after throwing an instance of 'std::regex_error' Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.854+0000 7f07659fc700 0 [progress INFO root] Loaded OSDMap, ready. Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: what(): Invalid start of range in bracket expression. Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: *** Caught signal (Aborted) ** Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: in thread 7f0768201700 thread_name:devicehealth Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 2: gsignal() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 3: abort() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa lse>&)+0x578) [0x560cee89a0a8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 27: pysqlite_connection_init() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 29: PyObject_Call() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: debug 2022-07-27T16:07:27.858+0000 7f0768201700 -1 *** Caught signal (Aborted) ** Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: in thread 7f0768201700 thread_name:devicehealth Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 2: gsignal() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 3: abort() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa lse>&)+0x578) [0x560cee89a0a8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 27: pysqlite_connection_init() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 29: PyObject_Call() Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1] Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 27 16:07:27 02.ceph-kubernetes.dev bash[1365938]: --- begin dump of recent events --- ... Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: debug 0> 2022-07-27T16:07:27.858+0000 7f0768201700 -1 *** Caught signal (Aborted) ** Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: in thread 7f0768201700 thread_name:devicehealth Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f07aecd8ce0] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 2: gsignal() Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 3: abort() Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f07ae0d709b] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f07ae0dd53c] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f07ae0dd597] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f07ae0dd7f8] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x560cee88eeea] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, fa lse>&)+0x578) [0x560cee89a0a8] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x560cee89b6b6] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x560cee89f63b] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x560cee89f71a] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x560cee89f9e8] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x560cee89fca0] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x560cee89fc12] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x560cee89fdf0] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x560cee8a03cb] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f07b14c27ca] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f07b14cb486] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f07b1208f1c] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f07b12704c9] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 27: pysqlite_connection_init() Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f07b0d8afc6] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 29: PyObject_Call() Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f079f9fb1f5] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f07b0ded5f1] Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 27 16:07:28 02.ceph-kubernetes.dev bash[1365938]: --- logging levels ---
Updated by Laura Flores over 1 year ago
Hi @Andrew Thomas,
There is a thread about this issue on the ceph-users mailing list (subject is "[ceph-users] 17.2.2: all MGRs crashing in fresh cephadm install"). Here is some info about the issue in case you are interested in testing a fix:
This issue seems to be showing up in 17.2.2, details in
https://tracker.ceph.com/issues/55304. We are currently in the process
of validating the fix https://github.com/ceph/ceph/pull/47270 and
we'll try to expedite a quick fix.
In the meantime, we have builds/images of the dev version of the fix,
in case you want to give it a try.
https://shaman.ceph.com/builds/ceph/wip-quincy-libcephsqlite-fix/
quay.ceph.io/ceph-ci/ceph:f516549e3e4815795ff0343ab71b3ebf567e5531
Updated by Neha Ojha over 1 year ago
Relevant thread in ceph-users https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5MTRTW2XDHMMU3TFBAA7AWYWXXI7XHQ2/
Updated by Telemetry Bot over 1 year ago
- Crash signature (v1) updated (diff)
- Crash signature (v2) updated (diff)
- Affected Versions v17.2.2 added
Sanitized backtrace:
std::__throw_regex_error(std::regex_constants::error_type, char const*)
bool std::__detail::_Compiler<std::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::regex_traits<char>, false, false>&)
void std::__detail::_Compiler<std::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)
pysqlite_connection_init()
PyObject_Call()
Crash dump sample:
{
"backtrace": [
"/lib64/libpthread.so.0(+0x12ce0) [0x7f95c2b3bce0]",
"gsignal()",
"abort()",
"/lib64/libstdc++.so.6(+0x9009b) [0x7f95c1f3a09b]",
"/lib64/libstdc++.so.6(+0x9653c) [0x7f95c1f4053c]",
"/lib64/libstdc++.so.6(+0x96597) [0x7f95c1f40597]",
"/lib64/libstdc++.so.6(+0x967f8) [0x7f95c1f407f8]",
"(std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x5641b965deea]",
"(bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x5641b96690a8]",
"(void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x5641b966a6b6]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x5641b966e63b]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x5641b966e71a]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x5641b966eca0]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x5641b966edf0]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x5641b966e9e8]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x5641b966eca0]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x5641b966ec12]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x5641b966edf0]",
"(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x5641b966f3cb]",
"/lib64/libcephsqlite.so(+0x1b7ca) [0x7f95c53257ca]",
"/lib64/libcephsqlite.so(+0x24486) [0x7f95c532e486]",
"/lib64/libsqlite3.so.0(+0x75f1c) [0x7f95c506bf1c]",
"/lib64/libsqlite3.so.0(+0xdd4c9) [0x7f95c50d34c9]",
"pysqlite_connection_init()",
"/lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f95c4bedfc6]",
"PyObject_Call()",
"/usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f95b385e1f5]",
"/lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f95c4c505f1]"
],
"ceph_version": "17.2.2",
"crash_id": "2022-07-25T23:51:19.181675Z_55d26f06-f310-4036-9b47-6019814ac322",
"entity_name": "mgr.d3c45872f563d8b9ab59b23d8f89dbfe9fc3806e",
"os_id": "centos",
"os_name": "CentOS Stream",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mgr",
"stack_sig": "c378729b55506b4208e12e7c1fa4fc14e459efd2f9c2c9e59518f9e3675f5b12",
"timestamp": "2022-07-25T23:51:19.181675Z",
"utsname_machine": "x86_64",
"utsname_release": "3.10.0-1160.36.2.el7.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Wed Jul 21 11:57:15 UTC 2021"
}Updated by Laura Flores over 1 year ago
- Crash signature (v1) updated (diff)
We ran a teuthology reproducer on 17.2.1 and 17.2.2 respectively:
17.2.1
https://shaman.ceph.com/builds/ceph/quincy/ec95624474b1871a821a912b8c3af68f8f8e7aa1/
http://pulpito.front.sepia.ceph.com/lflores-2022-07-28_14:26:31-fs:thrash:workloads-quincy-distro-default-smithi/
17.2.2
https://shaman.ceph.com/builds/ceph/quincy/b6e46b8939c67a6cc754abb4d0ece3c8918eccc3/
http://pulpito.front.sepia.ceph.com/lflores-2022-07-28_14:29:53-fs:thrash:workloads-quincy-distro-default-smithi/
From the 17.2.1 build:
Processing files: libcephsqlite-17.2.1-0.el8.x86_64
Provides: libcephsqlite = 2:17.2.1-0.el8 libcephsqlite(x86-64) = 2:17.2.1-0.el8 libcephsqlite.so()(64bit)
From the 17.2.2 build:
Processing files: libcephsqlite-17.2.2-0.el8.x86_64
warning: Ignoring invalid regex ^%{_scl_prefix}/.*|%{_root_sysconfdir}/rpm/macros.gcc-toolset-11-config$
warning: Ignoring invalid regex ^%{_root_sysconfdir}/rpm/macros.gcc-toolset-11-config$
Provides: libcephsqlite = 2:17.2.2-0.el8 libcephsqlite(x86-64) = 2:17.2.2-0.el8 libcephsqlite.so()(64bit)
We found the issue to be that 17.2.1 was built using gcc 8.5.0-13, while 17.2.2 used 8.5.0-14.
https://bugzilla.redhat.com/show_bug.cgi?id=2001788 is the only difference.
See the changelong here: https://centos.pkgs.org/8-stream/centos-appstream-x86_64/gcc-8.5.0-14.el8.x86_64.rpm.html
This bug was not caught on the RC because the RC build likely got built using gcc 8.5.0-13. The build is gone now though, so there is no way to tell for sure.
Updated by Laura Flores over 1 year ago
- Related to Tasks #56952: Set mgr_pool to true for a handful of tests in the rados qa suite added
Updated by Laura Flores over 1 year ago
Associated BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2110797
Updated by Brian Woods over 1 year ago
I may also be seeing this issue, but on a completely fresh deployment somehow.
https://tracker.ceph.com/issues/56745?next_issue_id=56971
Going to zero everything again and do one more clean deployment just to be 100% sure.
Updated by Laura Flores over 1 year ago
Hi @Brian, looks like you are indeed hitting this same issue. This crash occurs on 17.2.2 clusters due to a change in gcc version (at build time). There will be a hotfix release available soon to address the issue. See https://ceph.io/en/news/blog/2022/v17-2-3-quincy-released/ for more details.
Updated by Brian Woods over 1 year ago
@Laura, all good! Seems I had some crazy timing of some sort (maybe the cephadm on GIT hasn't been updated???). I had intended to deploy 17.2.3 from the beginning, but somehow got 17.2.2 but then deployed 17.2.3.... IDK How really. SO I am all good, and if that is the case, I am going to cancel my ticket as a duplicate.
Updated by Konstantin Shalygin over 1 year ago
- Has duplicate Bug #56745: unrecognized option ssl_server_port appears to cause failure of MGR added
Updated by Yuri Weinstein over 1 year ago
Updated by Andrew Davidoff over 1 year ago
Thanks @Laura! Apologies that I didn't see your updates until just now, otherwise I would have tested as you suggested.
Updated by Radoslaw Zarzynski over 1 year ago
- Has duplicate Bug #56726: Ceph Mgr crash due to missing OSD service added
Updated by Konstantin Shalygin over 1 year ago
- Status changed from Pending Backport to Resolved
Updated by Konstantin Shalygin over 1 year ago
- Tags deleted (
backport_processed)