Project

General

Profile

Actions
Copy link

Bug #56726

closed

Ceph Mgr crash due to missing OSD service

Added by Tejas C over 1 year ago. Updated over 1 year ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v17.2.2
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

on version /]# ceph -v
ceph version 17.2.2-253-g68c231f3 (68c231f3a14bc2e200f16597d1764093027c9b46) quincy (stable)

Post bootstrap of cluster 1 OSD had entry in crush , but no "systemctl" service entry was created for this OSD.

-7 0.05846 host ceph-tj-pri-skkxcy-node5
2 hdd 0.01949 osd.2 up 1.00000 1.00000
5 hdd 0.01949 osd.5 up 1.00000 1.00000
11 hdd 0.01949 osd.11 up 1.00000 1.00000
8 0 osd.8 down 0 1.00000

so the Mgr crashed when any "ceph orch" command was executed, and stayed down. *

Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  in thread 7f0de90aa700 thread_name:devicehealth
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  ceph version 17.2.2-253-g68c231f3 (68c231f3a14bc2e200f16597d1764093027c9b46) quincy (stable)
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  1: /lib64/libpthread.so.0(0x12ce0) [0x7f0e2f144ce0]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  2: gsignal()
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  3: abort()
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  4: /lib64/libstdc+.so.6(0x9009b) [0x7f0e2e54309b]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  5: /lib64/libstdc+.so.6(0x9653c) [0x7f0e2e54953c]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  6: /lib64/libstdc+.so.6(0x96597) [0x7f0e2e549597]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  7: /lib64/libstdc+.so.6(+0x967f8) [0x7f0e2e5497f8]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x55a4d7b1807a]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x55a4d7b23238]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x55a4d7b24846]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x55a4d7b287cb]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x55a4d7b288aa]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55a4d7b28e30]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55a4d7b28f80]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x55a4d7b28b78]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x55a4d7b28e30]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x55a4d7b28da2]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x55a4d7b28f80]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x55a4d7b2955b]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  23: /lib64/libcephsqlite.so(+0x1b7ea) [0x7f0e319307ea]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  24: /lib64/libcephsqlite.so(+0x244a6) [0x7f0e319394a6]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f0e31676f1c]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f0e316de4c9]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  27: pysqlite_connection_init()
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f0e311f8fc6]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  29: PyObject_Call()
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f0e1fe661f5]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f0e3125b5f1]
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:  NOTE: a copy of the executable, or `objdump rdS <executable>` is needed to interpret this.
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer-tmbptv[33666]:
    Jul 26 10:05:08 ceph-tj-pri-skkxcy-node1-installer ceph-e07e9488-0cc9-11ed-a42b-fa163e26d9ec-mgr-ceph-tj-pri-skkxcy-node1-installer*tmbptv[33666]:  -1025> 2022-07-26T14:05:08.867+0000 7f0de285d700 -1 mgr get_metadata_python Requested missing service osd.8

After removing this OSD from the cluster , and restarting the Mgr system entry , it was back up

  • mgr: ceph-tj-pri-skkxcy-node1-installer.tmbptv(active, since 13s)*

Related issues 1 (0 open — 1 closed)

Is duplicate of cephsqlite - Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operatorResolved

Actions
Actions
Copy link
 #1

Updated by Ilya Dryomov over 1 year ago

  • Target version deleted (v17.2.3)
Actions
Copy link
 #2

Updated by Radoslaw Zarzynski over 1 year ago

  • Status changed from New to Duplicate
Actions
Copy link
 #3

Updated by Radoslaw Zarzynski over 1 year ago

  • Is duplicate of Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added
Actions
Copy link

Also available in: Atom PDF