Actions
Bug #38204
closedBug #38094: mgr: crash list
mgr: segfault, prometheus, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Regular vstart.sh. CentOS7. Containerized. e0cc3d30ecf4678708a2d4781897daf4c50a92a9.
-8> 2019-02-06 11:41:17.357 7f86e0442700 10 mgr list_servers_python > -7> 2019-02-06 11:41:17.358 7f86e0442700 20 mgr[prometheus] mdsmap: {'session_autoclose': 300L, 'balancer': '', 'modified': '2019-02-06 11 -6> 2019-02-06 11:41:17.359 7f86e0442700 10 mgr list_servers_python > -5> 2019-02-06 11:41:17.364 7f86e0442700 10 mgr list_servers_python > -4> 2019-02-06 11:41:17.365 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 0, skipping occupation record for this osd -3> 2019-02-06 11:41:17.366 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 1, skipping occupation record for this osd -2> 2019-02-06 11:41:17.366 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 2, skipping occupation record for this osd -1> 2019-02-06 11:41:17.369 7f86e0442700 10 mgr list_servers_python > 0> 2019-02-06 11:41:17.418 7f86e0442700 -1 *** Caught signal (Segmentation fault) ** in thread 7f86e0442700 thread_name:prometheus ceph version Development (no_version) nautilus (dev) 1: (()+0x94e5f8) [0x55da2afda5f8] 2: (()+0xf5d0) [0x7f87052e45d0] 3: (PyFormatter::dump_pyobject(char const*, _object*)+0x30) [0x55da2ae2b378] 4: (PyFormatter::open_object_section(char const*)+0x30) [0x55da2ae2afbc] 5: (ActivePyModules::get_perf_schema_python(std::string const&, std::string const&)+0x456) [0x55da2ad0acb2] 6: (()+0x69e752) [0x55da2ad2a752] 7: (PyEval_EvalFrameEx()+0x6df0) [0x7f87076f1cf0] 8: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 9: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 10: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c] 11: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 12: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 13: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c] 14: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 15: (()+0x70a6d) [0x7f870767da6d] 16: (PyObject_Call()+0x43) [0x7f8707658a63] 17: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd] 18: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 19: (()+0x70a6d) [0x7f870767da6d] 20: (PyObject_Call()+0x43) [0x7f8707658a63] 21: (()+0x5aa55) [0x7f8707667a55] 22: (PyObject_Call()+0x43) [0x7f8707658a63] 23: (()+0xa2e27) [0x7f87076afe27] 24: (PyObject_Call()+0x43) [0x7f8707658a63] 25: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd] 26: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 27: (()+0x70978) [0x7f870767d978] 28: (PyObject_Call()+0x43) [0x7f8707658a63] 29: (()+0x5aa55) [0x7f8707667a55] 30: (PyObject_Call()+0x43) [0x7f8707658a63] 31: (()+0x56df8) [0x7f8707663df8] 32: (PyObject_Call()+0x43) [0x7f8707658a63] 33: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 34: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 35: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 36: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 37: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 38: (()+0x70978) [0x7f870767d978] 39: (PyObject_Call()+0x43) [0x7f8707658a63] 40: (()+0x5aa55) [0x7f8707667a55] 41: (PyObject_Call()+0x43) [0x7f8707658a63] 42: (()+0xa2a87) [0x7f87076afa87] 43: (()+0xa179f) [0x7f87076ae79f] 44: (PyObject_Call()+0x43) [0x7f8707658a63] 45: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 46: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 47: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 48: (()+0x70a6d) [0x7f870767da6d] 49: (PyObject_Call()+0x43) [0x7f8707658a63] 50: (()+0x5aa55) [0x7f8707667a55] 51: (PyObject_Call()+0x43) [0x7f8707658a63] 52: (()+0xa2e27) [0x7f87076afe27] 53: (PyObject_Call()+0x43) [0x7f8707658a63] 54: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd] 55: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 56: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c] 57: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 58: (()+0x70978) [0x7f870767d978] 59: (PyObject_Call()+0x43) [0x7f8707658a63] 60: (()+0x5aa55) [0x7f8707667a55] 61: (PyObject_Call()+0x43) [0x7f8707658a63] 62: (()+0xa2a87) [0x7f87076afa87] 63: (()+0xa179f) [0x7f87076ae79f] 64: (PyObject_Call()+0x43) [0x7f8707658a63] 65: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 66: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 67: (()+0x70978) [0x7f870767d978] 68: (PyObject_Call()+0x43) [0x7f8707658a63] 69: (()+0x5aa55) [0x7f8707667a55] 70: (PyObject_Call()+0x43) [0x7f8707658a63] 71: (()+0xa2e27) [0x7f87076afe27] 72: (PyObject_Call()+0x43) [0x7f8707658a63] 73: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 74: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 75: (()+0x70978) [0x7f870767d978] 76: (PyObject_Call()+0x43) [0x7f8707658a63] 77: (()+0x5aa55) [0x7f8707667a55] 78: (PyObject_Call()+0x43) [0x7f8707658a63] 79: (()+0xa2e27) [0x7f87076afe27] 80: (PyObject_Call()+0x43) [0x7f8707658a63] 81: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 82: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 83: (()+0x70978) [0x7f870767d978] 84: (PyObject_Call()+0x43) [0x7f8707658a63] 85: (()+0x5aa55) [0x7f8707667a55] 86: (PyObject_Call()+0x43) [0x7f8707658a63] 87: (()+0xa2e27) [0x7f87076afe27] 88: (PyObject_Call()+0x43) [0x7f8707658a63] 89: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 90: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d] 91: (()+0x70978) [0x7f870767d978] 92: (PyObject_Call()+0x43) [0x7f8707658a63] 93: (()+0x5aa55) [0x7f8707667a55] 94: (PyObject_Call()+0x43) [0x7f8707658a63] 95: (()+0xa2e27) [0x7f87076afe27] 96: (PyObject_Call()+0x43) [0x7f8707658a63] 97: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236] 98: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] 99: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Ernesto Puerta about 5 years ago
- Priority changed from High to Urgent
- Severity changed from 2 - major to 1 - critical
Another occurrence of this one (master de98f2e0d9):
-3> 2019-02-07 19:30:03.305 7fbee9ede700 10 mgr list_servers_python > -2> 2019-02-07 19:30:03.305 7fbee9ede700 20 mgr[prometheus] mdsmap: {'session_autoclose': 300L, 'balancer': '', 'modified': '2019-02-07 1 -1> 2019-02-07 19:30:03.306 7fbee9ede700 10 mgr list_servers_python > 0> 2019-02-07 19:30:03.331 7fbee9ede700 -1 *** Caught signal (Segmentation fault) ** in thread 7fbee9ede700 thread_name:prometheus ceph version Development (no_version) nautilus (dev) 1: (()+0x94e5f8) [0x55a8ea4515f8] 2: (()+0xf5d0) [0x7fbf0eb405d0] 3: (PyFormatter::dump_pyobject(char const*, _object*)+0x30) [0x55a8ea2a2378] 4: (PyFormatter::open_object_section(char const*)+0x30) [0x55a8ea2a1fbc] 5: (OSDMap::dump(ceph::Formatter*) const+0x7c8) [0x7fbf128c2776] 6: (()+0x678907) [0x55a8ea17b907] 7: (()+0x681f22) [0x55a8ea184f22] 8: (()+0x681065) [0x55a8ea184065] 9: (ActivePyModules::get_python(std::string const&)+0x20a) [0x55a8ea17cae4] 10: (()+0x69c0ff) [0x55a8ea19f0ff] 11: (PyEval_EvalFrameEx()+0x6df0) [0x7fbf10f4dcf0] 12: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 13: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 14: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c] 15: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 16: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 17: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c] 18: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 19: (()+0x70a6d) [0x7fbf10ed9a6d] 20: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 21: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd] 22: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 23: (()+0x70a6d) [0x7fbf10ed9a6d] 24: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 25: (()+0x5aa55) [0x7fbf10ec3a55] 26: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 27: (()+0xa2e27) [0x7fbf10f0be27] 28: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 29: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd] 30: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 31: (()+0x70978) [0x7fbf10ed9978] 32: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 33: (()+0x5aa55) [0x7fbf10ec3a55] 34: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 35: (()+0x56df8) [0x7fbf10ebfdf8] 36: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 37: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 38: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 39: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 40: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 41: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 42: (()+0x70978) [0x7fbf10ed9978] 43: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 44: (()+0x5aa55) [0x7fbf10ec3a55] 45: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 46: (()+0xa2a87) [0x7fbf10f0ba87] 47: (()+0xa179f) [0x7fbf10f0a79f] 48: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 49: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 50: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd] 51: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 52: (()+0x70a6d) [0x7fbf10ed9a6d] 53: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 54: (()+0x5aa55) [0x7fbf10ec3a55] 55: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 56: (()+0xa2e27) [0x7fbf10f0be27] 57: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 58: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd] 59: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 60: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c] 61: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 62: (()+0x70978) [0x7fbf10ed9978] 63: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 64: (()+0x5aa55) [0x7fbf10ec3a55] 65: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 66: (()+0xa2a87) [0x7fbf10f0ba87] 67: (()+0xa179f) [0x7fbf10f0a79f] 68: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 69: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 70: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 71: (()+0x70978) [0x7fbf10ed9978] 72: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 73: (()+0x5aa55) [0x7fbf10ec3a55] 74: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 75: (()+0xa2e27) [0x7fbf10f0be27] 76: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 77: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 78: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 79: (()+0x70978) [0x7fbf10ed9978] 80: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 81: (()+0x5aa55) [0x7fbf10ec3a55] 82: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 83: (()+0xa2e27) [0x7fbf10f0be27] 84: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 85: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 86: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 87: (()+0x70978) [0x7fbf10ed9978] 88: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 89: (()+0x5aa55) [0x7fbf10ec3a55] 90: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 91: (()+0xa2e27) [0x7fbf10f0be27] 92: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 93: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236] 94: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d] 95: (()+0x70978) [0x7fbf10ed9978] 96: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 97: (()+0x5aa55) [0x7fbf10ec3a55] 98: (PyObject_Call()+0x43) [0x7fbf10eb4a63] 99: (()+0xa2e27) [0x7fbf10f0be27]
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segfault, thread_name:prometheus, ActivePyModules::get_perf_schema_python() > PyFormatter::dump_pyobject() to mgr: segfault, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Updated by Sage Weil about 5 years ago
please link the path or url for the run for these
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segfault, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject to mgr: segfault, prometheus, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Updated by Ernesto Puerta about 5 years ago
Sage Weil wrote:
please link the path or url for the run for these
These runs are on my laptop + ceph-dev building/testing environment (CentOS7 default make). Anyway, I'm currently rebuilding from scratch just to ensure it has nothing to do with that.
Updated by Ernesto Puerta about 5 years ago
- Status changed from New to Fix Under Review
Potential fix at https://github.com/ceph/ceph/pull/26445
Updated by Sage Weil about 5 years ago
- Status changed from Fix Under Review to Resolved
Actions