Backport #44868
Updated by Nathan Cutler about 4 years ago
https://github.com/ceph/ceph/pull/34326 My environment: [root@gz-ceph-52-205 ceph]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@gz-ceph-52-205 ceph]# ceph -v ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) i built a monitor system by prometheus,now it seem ceph-mgr's prometheus module got troubles: 2018-07-18 05:56:20.006383 7fdc4d3a6700 -1 *** Caught signal (Segmentation fault) ** in thread 7fdc4d3a6700 thread_name:prometheus ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) 1: (()+0x3f0c81) [0x7fdc6f014c81] 2: (()+0xf100) [0x7fdc6c7d3100] 3: (ceph::buffer::ptr::ptr(ceph::buffer::ptr const&)+0x17) [0x7fdc6f016777] 4: (std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> >::operator=(std::list<ceph::buffer::ptr, std::allocator<ceph::buffer::ptr> > const&)+0xe9) [0x7fdc6eef3a49] 5: (ceph::buffer::list::operator=(ceph::buffer::list const&)+0x16) [0x7fdc6eef3af6] 6: (ActivePyModules::get_python(std::string const&)+0x12d1) [0x7fdc6eef0411] 7: (()+0x2ea7cd) [0x7fdc6ef0e7cd] 8: (PyEval_EvalFrameEx()+0x4594) [0x7fdc6e71aaa4] 9: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 10: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 11: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 12: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 13: (()+0x6f05d) [0x7fdc6e6a905d] 14: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 15: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7] 16: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 17: (()+0x6f05d) [0x7fdc6e6a905d] 18: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 19: (()+0x590a5) [0x7fdc6e6930a5] 20: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 21: (()+0xa13f7) [0x7fdc6e6db3f7] 22: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 23: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7] 24: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 25: (()+0x6ef68) [0x7fdc6e6a8f68] 26: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 27: (()+0x590a5) [0x7fdc6e6930a5] 28: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 29: (()+0x55448) [0x7fdc6e68f448] 30: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 31: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 32: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 33: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 34: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 35: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 36: (()+0x6ef68) [0x7fdc6e6a8f68] 37: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 38: (()+0x590a5) [0x7fdc6e6930a5] 39: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 40: (()+0xa1057) [0x7fdc6e6db057] 41: (()+0x9fd6f) [0x7fdc6e6d9d6f] 42: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 43: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 44: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 45: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 46: (()+0x6f05d) [0x7fdc6e6a905d] 47: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 48: (()+0x590a5) [0x7fdc6e6930a5] 49: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 50: (()+0xa13f7) [0x7fdc6e6db3f7] 51: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 52: (PyEval_EvalFrameEx()+0xde7) [0x7fdc6e7172f7] 53: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 54: (PyEval_EvalFrameEx()+0x425f) [0x7fdc6e71a76f] 55: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 56: (()+0x6ef68) [0x7fdc6e6a8f68] 57: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 58: (()+0x590a5) [0x7fdc6e6930a5] 59: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 60: (()+0xa1057) [0x7fdc6e6db057] 61: (()+0x9fd6f) [0x7fdc6e6d9d6f] 62: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 63: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 64: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 65: (()+0x6ef68) [0x7fdc6e6a8f68] 66: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 67: (()+0x590a5) [0x7fdc6e6930a5] 68: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 69: (()+0xa13f7) [0x7fdc6e6db3f7] 70: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 71: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 72: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 73: (()+0x6ef68) [0x7fdc6e6a8f68] 74: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 75: (()+0x590a5) [0x7fdc6e6930a5] 76: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 77: (()+0xa13f7) [0x7fdc6e6db3f7] 78: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 79: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 80: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 81: (()+0x6ef68) [0x7fdc6e6a8f68] 82: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 83: (()+0x590a5) [0x7fdc6e6930a5] 84: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 85: (()+0xa13f7) [0x7fdc6e6db3f7] 86: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 87: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 88: (PyEval_EvalCodeEx()+0x7ed) [0x7fdc6e71c0bd] 89: (()+0x6ef68) [0x7fdc6e6a8f68] 90: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 91: (()+0x590a5) [0x7fdc6e6930a5] 92: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 93: (()+0xa13f7) [0x7fdc6e6db3f7] 94: (PyObject_Call()+0x43) [0x7fdc6e6840b3] 95: (PyEval_EvalFrameEx()+0x1d4c) [0x7fdc6e71825c] 96: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 97: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 98: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 99: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] 100: (PyEval_EvalFrameEx()+0x4350) [0x7fdc6e71a860] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Thanks to god,this cluster is not my online-environment,but it still scare me -.-