Bug #24995: mgr: prometheus Segmentation fault - mgr - Ceph

Actions

Copy link

Bug #24995

closed

mgr: prometheus Segmentation fault

Added by 伟杰谭 almost 6 years ago. Updated almost 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

prometheus module

Target version:

% Done:

Source:

Tags:

Backport:

octopus, nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v12.2.5

ceph-qa-suite:

Pull request ID:

34245

Crash signature (v1):

Crash signature (v2):

Description

My environment:
[root@gz-ceph-52-205 ceph]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@gz-ceph-52-205 ceph]# ceph -v
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)

i built a monitor system by prometheus,now it seem ceph-mgr's prometheus module got troubles:

2018-07-18 05:56:20.006383 7fdc4d3a6700 -1 ** Caught signal (Segmentation fault) *
in thread 7fdc4d3a6700 thread_name:prometheus

ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
 1: (()+0x3f0c81) [0x7fdc6f014c81]
 2: (()+0xf100) [0x7fdc6c7d3100]
 3: (ceph::buffer::ptr::ptr(ceph::buffer::ptr const&)+0x17) [0x7fdc6f016777]
 4: (std::list&lt;ceph::buffer::ptr, std::allocator&lt;ceph::buffer::ptr&gt; >::operator=(std::list&lt;ceph::buffer::ptr, std::allocator&lt;ceph::buffer::ptr&gt; > const&)+0xe9) [0x7fdc6eef3a49]
 5: (ceph::buffer::list::operator=(ceph::buffer::list const&)+0x16) [0x7fdc6eef3af6]
 6: (ActivePyModules::get_python(std::string const&)+0x12d1) [0x7fdc6eef0411]
 7: (()+0x2ea7cd) [0x7fdc6ef0e7cd]
 8: (PyEval_EvalFrameEx()+0x4594) [0x7fdc6e71aaa4]
 9: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 10: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 11: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 12: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 13: (()+0x6f05d) [0x7fdc6e6a905d]
 14: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 15: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7]
 16: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 17: (()+0x6f05d) [0x7fdc6e6a905d]
 18: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 19: (()+0x590a5) [0x7fdc6e6930a5]
 20: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 21: (()+0xa13f7) [0x7fdc6e6db3f7]
 22: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 23: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7]
 24: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 25: (()+0x6ef68) [0x7fdc6e6a8f68]
 26: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 27: (()+0x590a5) [0x7fdc6e6930a5]
 28: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 29: (()+0x55448) [0x7fdc6e68f448]
 30: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 31: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 32: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 33: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 34: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 35: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 36: (()+0x6ef68) [0x7fdc6e6a8f68]
 37: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 38: (()+0x590a5) [0x7fdc6e6930a5]
 39: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 40: (()+0xa1057) [0x7fdc6e6db057]
 41: (()+0x9fd6f) [0x7fdc6e6d9d6f]
 42: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 43: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 44: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 45: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 46: (()+0x6f05d) [0x7fdc6e6a905d]
 47: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 48: (()+0x590a5) [0x7fdc6e6930a5]
 49: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 50: (()+0xa13f7) [0x7fdc6e6db3f7]
 51: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 52: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7]
 53: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 54: (PyEval_EvalFrameEx()+0x425f) [0x7fdc6e71a76f]
 55: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 56: (()+0x6ef68) [0x7fdc6e6a8f68]
 57: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 58: (()+0x590a5) [0x7fdc6e6930a5]
 59: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 60: (()+0xa1057) [0x7fdc6e6db057]
 61: (()+0x9fd6f) [0x7fdc6e6d9d6f]
 62: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 63: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 64: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 65: (()+0x6ef68) [0x7fdc6e6a8f68]
 66: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 67: (()+0x590a5) [0x7fdc6e6930a5]
 68: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 69: (()+0xa13f7) [0x7fdc6e6db3f7]
 70: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 71: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 72: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 73: (()+0x6ef68) [0x7fdc6e6a8f68]
 74: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 75: (()+0x590a5) [0x7fdc6e6930a5]
 76: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 77: (()+0xa13f7) [0x7fdc6e6db3f7]
 78: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 79: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 80: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 81: (()+0x6ef68) [0x7fdc6e6a8f68]
 82: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 83: (()+0x590a5) [0x7fdc6e6930a5]
 84: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 85: (()+0xa13f7) [0x7fdc6e6db3f7]
 86: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 87: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 88: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd]
 89: (()+0x6ef68) [0x7fdc6e6a8f68]
 90: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 91: (()+0x590a5) [0x7fdc6e6930a5]
 92: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 93: (()+0xa13f7) [0x7fdc6e6db3f7]
 94: (PyObject_Call()+0x43) [0x7fdc6e6840b3]
 95: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c]
 96: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 97: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 98: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 99: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 100: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860]
 NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

Thanks to god,this cluster is not my online-environment,but it still scare me -.-

Related issues 3 (1 open — 2 closed)

Actions

Copy link

Updated by Patrick Donnelly almost 6 years ago

Project changed from Ceph to mgr
Subject changed from ceph-mgr:prometheus Segmentation fault to mgr: prometheus Segmentation fault
Category deleted (~~Monitor~~)

Actions

Copy link

Updated by John Spray over 5 years ago

Category set to prometheus module

Actions

Copy link

Updated by Jan Fajerski over 5 years ago

Is this still an issue with a more up-to-date luminous? Are logs still available from this seg fault?

Actions

Copy link

Updated by Jan Fajerski over 5 years ago

Assignee set to Jan Fajerski

Actions

Copy link

Updated by 伟杰谭 over 5 years ago

no,this issue never happend again.it work fine in ceph version 12.2.8.

Actions

Copy link

Updated by Jan Fajerski over 5 years ago

Status changed from New to Can't reproduce

Alright I'll close this then. Please re-open if it happens again and thanks for reporting.

Actions

Copy link

Updated by Brad Hubbard about 4 years ago

Status changed from Can't reproduce to Need More Info

Very similar crash seen in Nautilus 14.2.6 and 14.2.7 with crash signature 6e2dd8773419635df43a4717e1eb3b32bce34f69a172290bc222923a05764ccc Possible stack overflow?

stack:
         ()
        ceph::buffer::v14_2_0::ptr_node::cloner::operator()(ceph::buffer::v14_2_0::ptr_node const&)
        ceph::buffer::v14_2_0::list::buffers_t::clone_from(ceph::buffer::v14_2_0::list::buffers_t const&)
        ActivePyModules::get_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
        ()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalCodeEx()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        ()
        PyObject_Call()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()
        PyEval_EvalFrameEx()

Actions

Copy link

Updated by Sebastian Wagner about 4 years ago

   -33> 2020-03-10 02:09:20.009 7f6912953700  4 mgrc reconnect Starting new session with v1:10.0.0.1:6808/23561
   -32> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_open from 0x55cfb6bfb680  client,dl380o-201
   -31> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_open from 0x55cfb9793180  client,dl380o-201
   -30> 2020-03-10 02:09:20.009 7f6914957700  4 mgrc handle_mgr_configure stats_period=5
   -29> 2020-03-10 02:09:20.009 7f690a848700  4 mgrc handle_mgr_configure stats_period=5
   -28> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_report from 0x55cfb6bfb680 client,dl380o-201
   -27> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_report rejecting report from non-daemon client dl380o-201
   -26> 2020-03-10 02:09:20.009 7f6914957700  4 mgrc ms_handle_reset ms_handle_reset con 0x55cf9f88c900
   -25> 2020-03-10 02:09:20.009 7f6914957700  4 mgrc reconnect Terminating session with v1:10.0.0.1:6808/23561
   -24> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_report from 0x55cfb9793180 client,dl380o-201
   -23> 2020-03-10 02:09:20.009 7f691b1a4700  4 mgr.server handle_report rejecting report from non-daemon client dl380o-201
   -22> 2020-03-10 02:09:20.009 7f6914957700  4 mgrc reconnect waiting to retry connect until 2020-03-10 02:09:21.013982
   -21> 2020-03-10 02:09:20.009 7f690a848700  4 mgrc ms_handle_reset ms_handle_reset con 0x55cf9f112d00
   -20> 2020-03-10 02:09:20.009 7f690a848700  4 mgrc reconnect Terminating session with v1:10.0.0.1:6808/23561
   -19> 2020-03-10 02:09:20.009 7f690a848700  4 mgrc reconnect waiting to retry connect until 2020-03-10 02:09:21.013980
   -18> 2020-03-10 02:09:20.565 7f692cd89700  4 mgr send_beacon active
   -17> 2020-03-10 02:09:20.565 7f692cd89700 10 monclient: _send_mon_message to mon.dl380o-201 at v2:10.0.0.1:3300/0
   -16> 2020-03-10 02:09:20.681 7f691b1a4700  4 mgr.server handle_report from 0x55cfb9e2d680 mds,prj-lb1appcephmds
   -15> 2020-03-10 02:09:20.789 7f6914957700  4 mgrc handle_mgr_map Got map version 1344138
   -14> 2020-03-10 02:09:20.789 7f6914957700  4 mgrc handle_mgr_map Active mgr is now v1:10.0.0.1:6808/23561
   -13> 2020-03-10 02:09:20.789 7f6914957700  4 mgrc reconnect waiting to retry connect until 2020-03-10 02:09:21.013982
   -12> 2020-03-10 02:09:20.789 7f6930590700  4 mgr ms_dispatch active mgrmap(e 1344138) v1
   -11> 2020-03-10 02:09:20.789 7f6930590700  4 mgr handle_mgr_map received map epoch 1344138
   -10> 2020-03-10 02:09:20.789 7f6930590700  4 mgr handle_mgr_map active in map: 1 active is 13601269
    -9> 2020-03-10 02:09:20.793 7f6930590700  4 mgr ms_dispatch mgrmap(e 1344138) v1
    -8> 2020-03-10 02:09:20.793 7f6930590700  4 mgrc handle_mgr_map Got map version 1344138
    -7> 2020-03-10 02:09:20.793 7f6930590700  4 mgrc handle_mgr_map Active mgr is now v1:10.0.0.1:6808/23561
    -6> 2020-03-10 02:09:20.793 7f690a848700  4 mgrc handle_mgr_map Got map version 1344138
    -5> 2020-03-10 02:09:20.793 7f690a848700  4 mgrc handle_mgr_map Active mgr is now v1:10.0.0.1:6808/23561
    -4> 2020-03-10 02:09:20.793 7f690a848700  4 mgrc reconnect waiting to retry connect until 2020-03-10 02:09:21.013980
    -3> 2020-03-10 02:09:20.829 7f691b1a4700  4 mgr.server handle_report from 0x55cfae7c1180 osd,11
    -2> 2020-03-10 02:09:20.941 7f6930590700  4 mgr ms_dispatch active mgrdigest v1
    -1> 2020-03-10 02:09:20.941 7f6930590700  4 mgr ms_dispatch mgrdigest v1
     0> 2020-03-10 02:09:20.945 7f6903ffb700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f6903ffb700 thread_name:prometheus

 ceph version 14.2.5-380-g1387ceaf78 (1387ceaf78464518975cb8efb2da92b07f7a4c9c) nautilus (stable)
 1: (()+0x12360) [0x7f693ad9a360]
 2: (ceph::buffer::v14_2_0::ptr_node::cloner::operator()(ceph::buffer::v14_2_0::ptr_node const&)+0x43) [0x7f693bd54c73]
 3: (ceph::buffer::v14_2_0::list::buffers_t::clone_from(ceph::buffer::v14_2_0::list::buffers_t const&)+0x8b) [0x55cf983cd91b]
 4: (ActivePyModules::get_python(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1c8c) [0x55cf983c4e7c]
 5: (()+0x1445fb) [0x55cf983d35fb]
 6: (_PyCFunction_FastCallDict()+0x8b) [0x7f693b34a99b]
 7: (()+0x182e4a) [0x7f693b39de4a]
 8: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 9: (()+0x1831da) [0x7f693b39e1da]
 10: (()+0x182f06) [0x7f693b39df06]
 11: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 12: (()+0x1831da) [0x7f693b39e1da]
 13: (()+0x182f06) [0x7f693b39df06]
 14: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 15: (()+0x1831da) [0x7f693b39e1da]
 16: (()+0x182f06) [0x7f693b39df06]
 17: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 18: (()+0x183451) [0x7f693b39e451]
 19: (()+0x182f06) [0x7f693b39df06]
 20: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 21: (_PyFunction_FastCallDict()+0x454) [0x7f693b39fc14]
 22: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 23: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 24: (PyObject_Call()+0x4b) [0x7f693b314aab]
 25: (_PyEval_EvalFrameDefault()+0x19e8) [0x7f693b398528]
 26: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 27: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 28: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 29: (PyObject_Call()+0x4b) [0x7f693b314aab]
 30: (()+0x143840) [0x7f693b35e840]
 31: (PyObject_Call()+0x4b) [0x7f693b314aab]
 32: (_PyEval_EvalFrameDefault()+0x19e8) [0x7f693b398528]
 33: (_PyFunction_FastCallDict()+0x454) [0x7f693b39fc14]
 34: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 35: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 36: (PyObject_Call()+0x4b) [0x7f693b314aab]
 37: (()+0x143840) [0x7f693b35e840]
 38: (_PyObject_FastCallDict()+0x8c) [0x7f693b31467c]
 39: (()+0x182f7d) [0x7f693b39df7d]
 40: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 41: (()+0x1831da) [0x7f693b39e1da]
 42: (()+0x182f06) [0x7f693b39df06]
 43: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 44: (()+0x1831da) [0x7f693b39e1da]
 45: (()+0x182f06) [0x7f693b39df06]
 46: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 47: (()+0x1831da) [0x7f693b39e1da]
 48: (()+0x182f06) [0x7f693b39df06]
 49: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 50: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 51: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 52: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 53: (PyObject_Call()+0x4b) [0x7f693b314aab]
 54: (()+0x1419cd) [0x7f693b35c9cd]
 55: (()+0x13f1fe) [0x7f693b35a1fe]
 56: (_PyObject_FastCallDict()+0x8c) [0x7f693b31467c]
 57: (()+0x182f7d) [0x7f693b39df7d]
 58: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 59: (()+0x1831da) [0x7f693b39e1da]
 60: (()+0x182f06) [0x7f693b39df06]
 61: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 62: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 63: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 64: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 65: (PyObject_Call()+0x4b) [0x7f693b314aab]
 66: (()+0x143840) [0x7f693b35e840]
 67: (PyObject_Call()+0x4b) [0x7f693b314aab]
 68: (_PyEval_EvalFrameDefault()+0x19e8) [0x7f693b398528]
 69: (()+0x183451) [0x7f693b39e451]
 70: (()+0x182f06) [0x7f693b39df06]
 71: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 72: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 73: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 74: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 75: (PyObject_Call()+0x4b) [0x7f693b314aab]
 76: (()+0x1419cd) [0x7f693b35c9cd]
 77: (()+0x13f1fe) [0x7f693b35a1fe]
 78: (_PyObject_FastCallDict()+0x8c) [0x7f693b31467c]
 79: (()+0x182f7d) [0x7f693b39df7d]
 80: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 81: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 82: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 83: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 84: (PyObject_Call()+0x4b) [0x7f693b314aab]
 85: (()+0x143840) [0x7f693b35e840]
 86: (_PyObject_FastCallDict()+0x8c) [0x7f693b31467c]
 87: (()+0x182f7d) [0x7f693b39df7d]
 88: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 89: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 90: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 91: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]
 92: (PyObject_Call()+0x4b) [0x7f693b314aab]
 93: (()+0x143840) [0x7f693b35e840]
 94: (_PyObject_FastCallDict()+0x8c) [0x7f693b31467c]
 95: (()+0x182f7d) [0x7f693b39df7d]
 96: (_PyEval_EvalFrameDefault()+0x2fa) [0x7f693b396e3a]
 97: (_PyFunction_FastCallDict()+0x12a) [0x7f693b39f8ea]
 98: (_PyObject_FastCallDict()+0x15e) [0x7f693b31474e]
 99: (_PyObject_Call_Prepend()+0x61) [0x7f693b314fe1]

Actions

Copy link

Updated by Jan Fajerski about 4 years ago

stack trace with a couple more symbols:

#0  raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x000055cf98570c70 in reraise_fatal (signum=11)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/global/signal_handler.cc:81
#2  handle_fatal_signal (signum=11) at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/global/signal_handler.cc:326
#3  <signal handler called>
#4  0x00007f693bd54c73 in ceph::buffer::v14_2_0::ptr_node::cloner::operator() (this=this@entry=0x7f6903ff3837, clone_this=...)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/common/buffer.cc:2212
#5  0x000055cf983cd91b in ceph::buffer::v14_2_0::list::buffers_t::clone_from (this=0x7f6903ff39e0, other=...)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/include/buffer.h:641
#6  0x000055cf983c4e7c in ceph::buffer::v14_2_0::list::operator= (other=..., this=0x7f6903ff39e0)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/include/buffer.h:1003
#7  ActivePyModules::get_python (this=this@entry=0x55cf9abbe780, what=...)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/mgr/ActivePyModules.cc:385
#8  0x000055cf983d35fb in ceph_state_get (self=<optimized out>, args=<optimized out>)
    at /usr/src/debug/ceph-14.2.5.380+g1387ceaf78-3.27.2.x86_64/src/mgr/BaseMgrModule.cc:353
#9  0x00007f693b34a99b in _PyCFunction_FastCallDict (func_obj=0x7f68f623fd80, args=0x7f68fcd2b548, nargs=<optimized out>, kwargs=0x0)
    at Objects/methodobject.c:234

Actions

Copy link

#10

Updated by Neha Ojha about 4 years ago

Related to Bug #44726: prometheus exporter crash added

Actions

Copy link

#11

Updated by Radoslaw Zarzynski about 4 years ago

Status changed from Need More Info to Fix Under Review
Assignee deleted (~~Jan Fajerski~~)

https://github.com/ceph/ceph/pull/34245

Actions

Copy link

#12

Updated by Jan Fajerski about 4 years ago

Backport set to octopus, nautilus
Pull request ID set to 34245

Actions

Copy link

#13

Updated by Radoslaw Zarzynski about 4 years ago

Copied to Backport #44868: nautilus: mgr: prometheus Segmentation fault added

Actions

Copy link

#14

Updated by Kefu Chai about 4 years ago

Status changed from Fix Under Review to Pending Backport

Actions

Copy link

#15

Updated by Konstantin Shalygin about 4 years ago

Copied to Backport #44986: octopus: mgr: prometheus Segmentation fault added

Actions

Copy link

#16

Updated by Nathan Cutler almost 4 years ago

Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » mgr

Custom queries

Bug #24995

mgr: prometheus Segmentation fault

Updated by Patrick Donnelly almost 6 years ago

Updated by John Spray over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by 伟杰谭 over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by Brad Hubbard about 4 years ago

Updated by Sebastian Wagner about 4 years ago

Updated by Jan Fajerski about 4 years ago

Updated by Neha Ojha about 4 years ago

Updated by Radoslaw Zarzynski about 4 years ago

Updated by Jan Fajerski about 4 years ago

Updated by Radoslaw Zarzynski about 4 years ago

Updated by Kefu Chai about 4 years ago

Updated by Konstantin Shalygin about 4 years ago

Updated by Nathan Cutler almost 4 years ago

Project

General

Profile

Ceph » mgr

Custom queries

Bug #24995

mgr: prometheus Segmentation fault

Updated by Patrick Donnelly almost 6 years ago

Updated by John Spray over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by 伟杰 谭 over 5 years ago

Updated by Jan Fajerski over 5 years ago

Updated by Brad Hubbard about 4 years ago

Updated by Sebastian Wagner about 4 years ago

Updated by Jan Fajerski about 4 years ago

Updated by Neha Ojha about 4 years ago

Updated by Radoslaw Zarzynski about 4 years ago

Updated by Jan Fajerski about 4 years ago

Updated by Radoslaw Zarzynski about 4 years ago

Updated by Kefu Chai about 4 years ago

Updated by Konstantin Shalygin about 4 years ago

Updated by Nathan Cutler almost 4 years ago

Updated by 伟杰谭 over 5 years ago