Bug #38112
closedBug #38094: mgr: crash list
mgr: segfault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc
0%
Description
Default vstart.sh cluster, running on RHEL 7. Ceph master (a9dfd7067f8bee78bf225bcbe484470aaa1a044f). During regular operation (no recent restart or module reload).
-2> 2019-01-30 12:07:10.715 7feac54dc700 20 mgr[dashboard] NQ: processing queue: 1
-1> 2019-01-30 12:07:10.715 7fead8c55700 20 mgr Gil Switched to new thread state 0x55e8ac941ce0
0> 2019-01-30 12:07:10.756 7fead8c55700 -1 *** Caught signal (Segmentation fault) **
in thread 7fead8c55700 thread_name:mgr-fin
ceph version Development (no_version) nautilus (dev)
1: (()+0x93c92a) [0x55e8a702992a]
2: (()+0xf5d0) [0x7feaf444e5d0]
3: (PyObject_Malloc()+0xa7) [0x7feaf6800ef7]
4: (PyString_FromString()+0x95) [0x7feaf6810b75]
5: (PyFormatter::finish_pending_streams()+0xc9) [0x55e8a6e7b063]
6: (PyFormatter::get()+0x18) [0x55e8a6d50e1c]
7: (ActivePyModules::get_python(std::string const&)+0x7c7) [0x55e8a6d561a7]
8: (()+0x68b43b) [0x55e8a6d7843b]
9: (PyEval_EvalFrameEx()+0x6df0) [0x7feaf685bcf0]
10: (PyEval_EvalFrameEx()+0x67bd) [0x7feaf685b6bd]
11: (PyEval_EvalCodeEx()+0x7ed) [0x7feaf685e03d]
12: (()+0x70978) [0x7feaf67e7978]
13: (PyObject_Call()+0x43) [0x7feaf67c2a63]
14: (()+0x5aa55) [0x7feaf67d1a55]
15: (PyObject_Call()+0x43) [0x7feaf67c2a63]
16: (()+0x4bb45) [0x7feaf67c2b45]
17: (PyObject_CallMethod()+0xbb) [0x7feaf67c2e7b]
18: (ActivePyModule::notify(std::string const&, std::string const&)+0xc6) [0x55e8a6d4f218]
19: (()+0x66a560) [0x55e8a6d57560]
20: (()+0x67176a) [0x55e8a6d5e76a]
21: (boost::function1<void, int>::operator()(int) const+0x6c) [0x55e8a6d63962]
22: (FunctionContext::finish(int)+0x24) [0x55e8a6d606ea]
23: (Context::complete(int)+0x27) [0x55e8a6d605b1]
24: (Finisher::finisher_thread_entry()+0x38b) [0x7feaf7bfd749]
25: (Finisher::FinisherThread::entry()+0x1c) [0x55e8a6e3fae4]
26: (Thread::entry_wrapper()+0x78) [0x7feaf7c6a42c]
27: (Thread::_entry_func(void*)+0x18) [0x7feaf7c6a3aa]
28: (()+0x7dd5) [0x7feaf4446dd5]
29: (clone()+0x6d) [0x7feaf30f6ead]
Updated by Ernesto Puerta about 5 years ago
The issue might be at PyFormatter.cc:PyFormatter::finish_pending_streams
void PyFormatter::finish_pending_streams()
{
for (const auto &i : pending_streams) {
PyObject *tmp_cur = cursor;
cursor = i->cursor;
dump_pyobject(
i->name.c_str(),
PyString_FromString(i->stream.str().c_str()));
cursor = tmp_cur;
}
pending_streams.clear();
}
Updated by Ernesto Puerta about 5 years ago
After looking at the code of `PyString_FromString`, the issue seems not to be related to the input `c_str()` pointer, but with the `PyObject_Malloc` itself. It's a memory allocator based on malloc for bulk reservations. Python3 has the option to change the memory allocator behaviour via env. variable, but Py2 does not seem to have such a thing. I'll install python-debug in order to display the missing symbols in the stack trace.
Updated by Volker Theile about 5 years ago
I've done some research related to PyObject_Malloc errors and found out that most of the forum comments i found came to the conclusion that mostly an incorrect ref counting is the reason for crashing a Python C extension. The problem is to identify this because the origin of the crash is not where the crash happens.
Updated by Volker Theile about 5 years ago
A good starting point could be https://github.com/ceph/ceph/blob/master/src/mgr/BaseMgrModule.cc#L405 because this is the method that can be found in the log file near every seg fault i came across.
Another one that should have been checked is https://github.com/ceph/ceph/blob/master/src/mgr/ActivePyModules.cc#L529.
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segmentation fault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc to mgr: segfault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc
Updated by Sage Weil about 5 years ago
- Status changed from New to Resolved