Project

General

Profile

Bug #56700

MGR pod on CLBO on rook deployment

Added by Parth Arora 2 months ago. Updated about 2 months ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Regression:
Yes
Severity:
1 - critical
Reviewed:
07/25/2022
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

MGR pod is failing for the new ceph version v17.2.2, tillv17.2.1 it was working fine.

```
29: PyObject_Call()
30: /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2fe1fd41f5]
31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2ff33c65f1]
debug 2022-07-25T08:28:25.405+0000 7f2f83a25700 -1 ** Caught signal (Aborted) *
```

It's quite urgent in the sense we need to release ODF4.11 and that would be linked to the latest ceph version so we quickly need a new ceph release out if it's really a problem.

References: https://github.com/rook/rook/issues/10627 (full logs)
Changes done: https://github.com/ceph/ceph/compare/v17.2.1...v17.2.2


Related issues

Duplicates cephsqlite - Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator Resolved

History

#1 Updated by Parth Arora 2 months ago

Parth Arora wrote:

MGR pod is failing for the new ceph version v17.2.2, till v17.2.1 it was working fine.

```
29: PyObject_Call()
30: /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2fe1fd41f5]
31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2ff33c65f1]
debug 2022-07-25T08:28:25.405+0000 7f2f83a25700 -1 ** Caught signal (Aborted) *
```

We quickly need a new ceph release out if it's really a problem.

References: https://github.com/rook/rook/issues/10627 (full logs)
Changes done: https://github.com/ceph/ceph/compare/v17.2.1...v17.2.2

#2 Updated by Vikhyat Umrao 2 months ago

  • Duplicates Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added

#3 Updated by Vikhyat Umrao 2 months ago

  • Status changed from New to Duplicate

#4 Updated by Neha Ojha 2 months ago

Quoting from a chat group:

@Travis Nielsen I think the issue you are seeing was first seen in https://tracker.ceph.com/issues/55696#note-2 and seems to have been resolved in https://tracker.ceph.com/issues/55304. But the patch is not present in Quincy yet, I just marked it with Pending backport.

#5 Updated by Laura Flores 2 months ago

I am hitting a bunch of these failures on a recent teuthology run I scheduled. The ceph version is 17.2.0:

http://pulpito.front.sepia.ceph.com/lflores-2022-07-25_21:46:45-fs:thrash:workloads-wip-telemetry-upgrade-distro-default-smithi/

2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr:terminate called after throwing an instance of 'std::regex_error'
2022-07-25T22:16:04.368 INFO:tasks.ceph.mgr.x.smithi188.stderr:  what():  Invalid start of range in bracket expression.
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr:*** Caught signal (Aborted) **
2022-07-25T22:16:04.369 INFO:tasks.ceph.mgr.x.smithi188.stderr: in thread 7f1922ff6700 thread_name:devicehealth
2022-07-25T22:16:04.370 INFO:tasks.ceph.mgr.x.smithi188.stderr: ceph version 17.2.0-661-ge6650b3c (e6650b3cd2b2626a7724085062c110eb4de431b5) quincy (stable)
2022-07-25T22:16:04.371 INFO:tasks.ceph.mgr.x.smithi188.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f198c818ce0]
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 2: gsignal()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 3: abort()
2022-07-25T22:16:04.372 INFO:tasks.ceph.mgr.x.smithi188.stderr: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f198bc1709b]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f198bc1d53c]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f198bc1d597]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f198bc1d7f8]
2022-07-25T22:16:04.373 INFO:tasks.ceph.mgr.x.smithi188.stderr: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x55d1dc419f5a]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x55d1dc425118]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x55d1dc426726]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x55d1dc42a6ab]
2022-07-25T22:16:04.374 INFO:tasks.ceph.mgr.x.smithi188.stderr: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x55d1dc42a78a]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x55d1dc42aa58]
2022-07-25T22:16:04.375 INFO:tasks.ceph.mgr.x.smithi188.stderr: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55d1dc42ad10]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.376 INFO:tasks.ceph.mgr.x.smithi188.stderr: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55d1dc42ac82]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55d1dc42ae60]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x55d1dc42b43b]
2022-07-25T22:16:04.377 INFO:tasks.ceph.mgr.x.smithi188.stderr: 23: /lib64/libcephsqlite.so(+0x1b7fa) [0x7f198f0047fa]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 24: /lib64/libcephsqlite.so(+0x244b6) [0x7f198f00d4b6]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f198ed4af1c]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f198edb24c9]
2022-07-25T22:16:04.378 INFO:tasks.ceph.mgr.x.smithi188.stderr: 27: pysqlite_connection_init()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f198e8ccfc6]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 29: PyObject_Call()
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 30: /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f197d6cd1f5]
2022-07-25T22:16:04.379 INFO:tasks.ceph.mgr.x.smithi188.stderr: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f198e92f5f1]

I will leave a note about these occurrences on the original Tracker as well.

#6 Updated by Ilya Dryomov about 2 months ago

  • Target version deleted (v17.2.3)

Also available in: Atom PDF