Actions
Bug #38204
closedBug #38094: mgr: crash list
mgr: segfault, prometheus, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Regular vstart.sh. CentOS7. Containerized. e0cc3d30ecf4678708a2d4781897daf4c50a92a9.
-8> 2019-02-06 11:41:17.357 7f86e0442700 10 mgr list_servers_python >
-7> 2019-02-06 11:41:17.358 7f86e0442700 20 mgr[prometheus] mdsmap: {'session_autoclose': 300L, 'balancer': '', 'modified': '2019-02-06 11
-6> 2019-02-06 11:41:17.359 7f86e0442700 10 mgr list_servers_python >
-5> 2019-02-06 11:41:17.364 7f86e0442700 10 mgr list_servers_python >
-4> 2019-02-06 11:41:17.365 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 0, skipping occupation record for this osd
-3> 2019-02-06 11:41:17.366 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 1, skipping occupation record for this osd
-2> 2019-02-06 11:41:17.366 7f86e0442700 4 mgr[prometheus] Missing dev node metadata for osd 2, skipping occupation record for this osd
-1> 2019-02-06 11:41:17.369 7f86e0442700 10 mgr list_servers_python >
0> 2019-02-06 11:41:17.418 7f86e0442700 -1 *** Caught signal (Segmentation fault) **
in thread 7f86e0442700 thread_name:prometheus
ceph version Development (no_version) nautilus (dev)
1: (()+0x94e5f8) [0x55da2afda5f8]
2: (()+0xf5d0) [0x7f87052e45d0]
3: (PyFormatter::dump_pyobject(char const*, _object*)+0x30) [0x55da2ae2b378]
4: (PyFormatter::open_object_section(char const*)+0x30) [0x55da2ae2afbc]
5: (ActivePyModules::get_perf_schema_python(std::string const&, std::string const&)+0x456) [0x55da2ad0acb2]
6: (()+0x69e752) [0x55da2ad2a752]
7: (PyEval_EvalFrameEx()+0x6df0) [0x7f87076f1cf0]
8: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
9: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
10: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c]
11: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
12: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
13: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c]
14: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
15: (()+0x70a6d) [0x7f870767da6d]
16: (PyObject_Call()+0x43) [0x7f8707658a63]
17: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd]
18: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
19: (()+0x70a6d) [0x7f870767da6d]
20: (PyObject_Call()+0x43) [0x7f8707658a63]
21: (()+0x5aa55) [0x7f8707667a55]
22: (PyObject_Call()+0x43) [0x7f8707658a63]
23: (()+0xa2e27) [0x7f87076afe27]
24: (PyObject_Call()+0x43) [0x7f8707658a63]
25: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd]
26: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
27: (()+0x70978) [0x7f870767d978]
28: (PyObject_Call()+0x43) [0x7f8707658a63]
29: (()+0x5aa55) [0x7f8707667a55]
30: (PyObject_Call()+0x43) [0x7f8707658a63]
31: (()+0x56df8) [0x7f8707663df8]
32: (PyObject_Call()+0x43) [0x7f8707658a63]
33: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
34: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
35: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
36: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
37: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
38: (()+0x70978) [0x7f870767d978]
39: (PyObject_Call()+0x43) [0x7f8707658a63]
40: (()+0x5aa55) [0x7f8707667a55]
41: (PyObject_Call()+0x43) [0x7f8707658a63]
42: (()+0xa2a87) [0x7f87076afa87]
43: (()+0xa179f) [0x7f87076ae79f]
44: (PyObject_Call()+0x43) [0x7f8707658a63]
45: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
46: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
47: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
48: (()+0x70a6d) [0x7f870767da6d]
49: (PyObject_Call()+0x43) [0x7f8707658a63]
50: (()+0x5aa55) [0x7f8707667a55]
51: (PyObject_Call()+0x43) [0x7f8707658a63]
52: (()+0xa2e27) [0x7f87076afe27]
53: (PyObject_Call()+0x43) [0x7f8707658a63]
54: (PyEval_EvalFrameEx()+0x17fd) [0x7f87076ec6fd]
55: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
56: (PyEval_EvalFrameEx()+0x663c) [0x7f87076f153c]
57: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
58: (()+0x70978) [0x7f870767d978]
59: (PyObject_Call()+0x43) [0x7f8707658a63]
60: (()+0x5aa55) [0x7f8707667a55]
61: (PyObject_Call()+0x43) [0x7f8707658a63]
62: (()+0xa2a87) [0x7f87076afa87]
63: (()+0xa179f) [0x7f87076ae79f]
64: (PyObject_Call()+0x43) [0x7f8707658a63]
65: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
66: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
67: (()+0x70978) [0x7f870767d978]
68: (PyObject_Call()+0x43) [0x7f8707658a63]
69: (()+0x5aa55) [0x7f8707667a55]
70: (PyObject_Call()+0x43) [0x7f8707658a63]
71: (()+0xa2e27) [0x7f87076afe27]
72: (PyObject_Call()+0x43) [0x7f8707658a63]
73: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
74: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
75: (()+0x70978) [0x7f870767d978]
76: (PyObject_Call()+0x43) [0x7f8707658a63]
77: (()+0x5aa55) [0x7f8707667a55]
78: (PyObject_Call()+0x43) [0x7f8707658a63]
79: (()+0xa2e27) [0x7f87076afe27]
80: (PyObject_Call()+0x43) [0x7f8707658a63]
81: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
82: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
83: (()+0x70978) [0x7f870767d978]
84: (PyObject_Call()+0x43) [0x7f8707658a63]
85: (()+0x5aa55) [0x7f8707667a55]
86: (PyObject_Call()+0x43) [0x7f8707658a63]
87: (()+0xa2e27) [0x7f87076afe27]
88: (PyObject_Call()+0x43) [0x7f8707658a63]
89: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
90: (PyEval_EvalCodeEx()+0x7ed) [0x7f87076f403d]
91: (()+0x70978) [0x7f870767d978]
92: (PyObject_Call()+0x43) [0x7f8707658a63]
93: (()+0x5aa55) [0x7f8707667a55]
94: (PyObject_Call()+0x43) [0x7f8707658a63]
95: (()+0xa2e27) [0x7f87076afe27]
96: (PyObject_Call()+0x43) [0x7f8707658a63]
97: (PyEval_EvalFrameEx()+0x2336) [0x7f87076ed236]
98: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
99: (PyEval_EvalFrameEx()+0x67bd) [0x7f87076f16bd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Ernesto Puerta about 5 years ago
- Priority changed from High to Urgent
- Severity changed from 2 - major to 1 - critical
Another occurrence of this one (master de98f2e0d9):
-3> 2019-02-07 19:30:03.305 7fbee9ede700 10 mgr list_servers_python >
-2> 2019-02-07 19:30:03.305 7fbee9ede700 20 mgr[prometheus] mdsmap: {'session_autoclose': 300L, 'balancer': '', 'modified': '2019-02-07 1
-1> 2019-02-07 19:30:03.306 7fbee9ede700 10 mgr list_servers_python >
0> 2019-02-07 19:30:03.331 7fbee9ede700 -1 *** Caught signal (Segmentation fault) **
in thread 7fbee9ede700 thread_name:prometheus
ceph version Development (no_version) nautilus (dev)
1: (()+0x94e5f8) [0x55a8ea4515f8]
2: (()+0xf5d0) [0x7fbf0eb405d0]
3: (PyFormatter::dump_pyobject(char const*, _object*)+0x30) [0x55a8ea2a2378]
4: (PyFormatter::open_object_section(char const*)+0x30) [0x55a8ea2a1fbc]
5: (OSDMap::dump(ceph::Formatter*) const+0x7c8) [0x7fbf128c2776]
6: (()+0x678907) [0x55a8ea17b907]
7: (()+0x681f22) [0x55a8ea184f22]
8: (()+0x681065) [0x55a8ea184065]
9: (ActivePyModules::get_python(std::string const&)+0x20a) [0x55a8ea17cae4]
10: (()+0x69c0ff) [0x55a8ea19f0ff]
11: (PyEval_EvalFrameEx()+0x6df0) [0x7fbf10f4dcf0]
12: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
13: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
14: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c]
15: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
16: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
17: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c]
18: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
19: (()+0x70a6d) [0x7fbf10ed9a6d]
20: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
21: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd]
22: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
23: (()+0x70a6d) [0x7fbf10ed9a6d]
24: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
25: (()+0x5aa55) [0x7fbf10ec3a55]
26: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
27: (()+0xa2e27) [0x7fbf10f0be27]
28: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
29: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd]
30: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
31: (()+0x70978) [0x7fbf10ed9978]
32: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
33: (()+0x5aa55) [0x7fbf10ec3a55]
34: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
35: (()+0x56df8) [0x7fbf10ebfdf8]
36: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
37: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
38: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
39: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
40: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
41: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
42: (()+0x70978) [0x7fbf10ed9978]
43: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
44: (()+0x5aa55) [0x7fbf10ec3a55]
45: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
46: (()+0xa2a87) [0x7fbf10f0ba87]
47: (()+0xa179f) [0x7fbf10f0a79f]
48: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
49: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
50: (PyEval_EvalFrameEx()+0x67bd) [0x7fbf10f4d6bd]
51: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
52: (()+0x70a6d) [0x7fbf10ed9a6d]
53: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
54: (()+0x5aa55) [0x7fbf10ec3a55]
55: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
56: (()+0xa2e27) [0x7fbf10f0be27]
57: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
58: (PyEval_EvalFrameEx()+0x17fd) [0x7fbf10f486fd]
59: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
60: (PyEval_EvalFrameEx()+0x663c) [0x7fbf10f4d53c]
61: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
62: (()+0x70978) [0x7fbf10ed9978]
63: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
64: (()+0x5aa55) [0x7fbf10ec3a55]
65: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
66: (()+0xa2a87) [0x7fbf10f0ba87]
67: (()+0xa179f) [0x7fbf10f0a79f]
68: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
69: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
70: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
71: (()+0x70978) [0x7fbf10ed9978]
72: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
73: (()+0x5aa55) [0x7fbf10ec3a55]
74: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
75: (()+0xa2e27) [0x7fbf10f0be27]
76: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
77: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
78: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
79: (()+0x70978) [0x7fbf10ed9978]
80: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
81: (()+0x5aa55) [0x7fbf10ec3a55]
82: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
83: (()+0xa2e27) [0x7fbf10f0be27]
84: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
85: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
86: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
87: (()+0x70978) [0x7fbf10ed9978]
88: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
89: (()+0x5aa55) [0x7fbf10ec3a55]
90: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
91: (()+0xa2e27) [0x7fbf10f0be27]
92: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
93: (PyEval_EvalFrameEx()+0x2336) [0x7fbf10f49236]
94: (PyEval_EvalCodeEx()+0x7ed) [0x7fbf10f5003d]
95: (()+0x70978) [0x7fbf10ed9978]
96: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
97: (()+0x5aa55) [0x7fbf10ec3a55]
98: (PyObject_Call()+0x43) [0x7fbf10eb4a63]
99: (()+0xa2e27) [0x7fbf10f0be27]
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segfault, thread_name:prometheus, ActivePyModules::get_perf_schema_python() > PyFormatter::dump_pyobject() to mgr: segfault, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Updated by Sage Weil about 5 years ago
please link the path or url for the run for these
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segfault, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject to mgr: segfault, prometheus, ActivePyModules::get_perf_schema_python > PyFormatter::dump_pyobject
Updated by Ernesto Puerta about 5 years ago
Sage Weil wrote:
please link the path or url for the run for these
These runs are on my laptop + ceph-dev building/testing environment (CentOS7 default make). Anyway, I'm currently rebuilding from scratch just to ensure it has nothing to do with that.
Updated by Ernesto Puerta about 5 years ago
- Status changed from New to Fix Under Review
Potential fix at https://github.com/ceph/ceph/pull/26445
Updated by Sage Weil about 5 years ago
- Status changed from Fix Under Review to Resolved
Actions