Project

General

Profile

Bug #56726

Ceph Mgr crash due to missing OSD service

Added by Tejas C 4 months ago. Updated 3 months ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

on version /]# ceph -v
ceph version 17.2.2-253-g68c231f3 (68c231f3a14bc2e200f16597d1764093027c9b46) quincy (stable)

Post bootstrap of cluster 1 OSD had entry in crush , but no "systemctl" service entry was created for this OSD.

-7 0.05846 host ceph-tj-pri-skkxcy-node5
2 hdd 0.01949 osd.2 up 1.00000 1.00000
5 hdd 0.01949 osd.5 up 1.00000 1.00000
11 hdd 0.01949 osd.11 up 1.00000 1.00000
8 0 osd.8 down 0 1.00000

so the Mgr crashed when any "ceph orch" command was executed, and stayed down. *

Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  in thread 7f0de90aa700 thread_name:devicehealth
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: ceph version 17.2.2-253-g68c231f3 (68c231f3a14bc2e200f16597d1764093027c9b46) quincy (stable)
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 1: /lib64/libpthread.so.0(0x12ce0) [0x7f0e2f144ce0]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 2: gsignal()
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 3: abort()
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 4: /lib64/libstdc
+.so.6(0x9009b) [0x7f0e2e54309b]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 5: /lib64/libstdc
+.so.6(0x9653c) [0x7f0e2e54953c]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 6: /lib64/libstdc
+.so.6(0x96597) [0x7f0e2e549597]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 7: /lib64/libstdc
+.so.6(+0x967f8) [0x7f0e2e5497f8]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x55a4d7b1807a]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x55a4d7b23238]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x55a4d7b24846]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x55a4d7b287cb]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x55a4d7b288aa]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55a4d7b28e30]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55a4d7b28f80]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x55a4d7b28b78]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55a4d7b28e30]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55a4d7b28f80]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x55a4d7b2955b]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 23: /lib64/libcephsqlite.so(+0x1b7ea) [0x7f0e319307ea]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 24: /lib64/libcephsqlite.so(+0x244a6) [0x7f0e319394a6]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f0e31676f1c]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f0e316de4c9]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 27: pysqlite_connection_init()
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f0e311f8fc6]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 29: PyObject_Call()
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f0e1fe661f5]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f0e3125b5f1]
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]: NOTE: a copy of the executable, or `objdump rdS <executable>` is needed to interpret this.
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:
Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer
*tmbptv[33666]: -1025> 2022-07-26T14:05:08.867+0000 7f0de285d700 -1 mgr get_metadata_python Requested missing service osd.8

After removing this OSD from the cluster , and restarting the Mgr system entry , it was back up

  • mgr: ceph-tj-pri-skkxcy-node1-installer.tmbptv(active, since 13s)*

Related issues

Duplicates cephsqlite - Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator Resolved

History

#1 Updated by Ilya Dryomov 4 months ago

  • Target version deleted (v17.2.3)

#2 Updated by Radoslaw Zarzynski 3 months ago

  • Status changed from New to Duplicate

#3 Updated by Radoslaw Zarzynski 3 months ago

  • Duplicates Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added

Also available in: Atom PDF