Bug #38112
closedBug #38094: mgr: crash list
mgr: segfault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc
0%
Description
Default vstart.sh cluster, running on RHEL 7. Ceph master (a9dfd7067f8bee78bf225bcbe484470aaa1a044f). During regular operation (no recent restart or module reload).
-2> 2019-01-30 12:07:10.715 7feac54dc700 20 mgr[dashboard] NQ: processing queue: 1 -1> 2019-01-30 12:07:10.715 7fead8c55700 20 mgr Gil Switched to new thread state 0x55e8ac941ce0 0> 2019-01-30 12:07:10.756 7fead8c55700 -1 *** Caught signal (Segmentation fault) ** in thread 7fead8c55700 thread_name:mgr-fin ceph version Development (no_version) nautilus (dev) 1: (()+0x93c92a) [0x55e8a702992a] 2: (()+0xf5d0) [0x7feaf444e5d0] 3: (PyObject_Malloc()+0xa7) [0x7feaf6800ef7] 4: (PyString_FromString()+0x95) [0x7feaf6810b75] 5: (PyFormatter::finish_pending_streams()+0xc9) [0x55e8a6e7b063] 6: (PyFormatter::get()+0x18) [0x55e8a6d50e1c] 7: (ActivePyModules::get_python(std::string const&)+0x7c7) [0x55e8a6d561a7] 8: (()+0x68b43b) [0x55e8a6d7843b] 9: (PyEval_EvalFrameEx()+0x6df0) [0x7feaf685bcf0] 10: (PyEval_EvalFrameEx()+0x67bd) [0x7feaf685b6bd] 11: (PyEval_EvalCodeEx()+0x7ed) [0x7feaf685e03d] 12: (()+0x70978) [0x7feaf67e7978] 13: (PyObject_Call()+0x43) [0x7feaf67c2a63] 14: (()+0x5aa55) [0x7feaf67d1a55] 15: (PyObject_Call()+0x43) [0x7feaf67c2a63] 16: (()+0x4bb45) [0x7feaf67c2b45] 17: (PyObject_CallMethod()+0xbb) [0x7feaf67c2e7b] 18: (ActivePyModule::notify(std::string const&, std::string const&)+0xc6) [0x55e8a6d4f218] 19: (()+0x66a560) [0x55e8a6d57560] 20: (()+0x67176a) [0x55e8a6d5e76a] 21: (boost::function1<void, int>::operator()(int) const+0x6c) [0x55e8a6d63962] 22: (FunctionContext::finish(int)+0x24) [0x55e8a6d606ea] 23: (Context::complete(int)+0x27) [0x55e8a6d605b1] 24: (Finisher::finisher_thread_entry()+0x38b) [0x7feaf7bfd749] 25: (Finisher::FinisherThread::entry()+0x1c) [0x55e8a6e3fae4] 26: (Thread::entry_wrapper()+0x78) [0x7feaf7c6a42c] 27: (Thread::_entry_func(void*)+0x18) [0x7feaf7c6a3aa] 28: (()+0x7dd5) [0x7feaf4446dd5] 29: (clone()+0x6d) [0x7feaf30f6ead]
Updated by Ernesto Puerta about 5 years ago
The issue might be at PyFormatter.cc:PyFormatter::finish_pending_streams
void PyFormatter::finish_pending_streams()
{
for (const auto &i : pending_streams) {
PyObject *tmp_cur = cursor;
cursor = i->cursor;
dump_pyobject(
i->name.c_str(),
PyString_FromString(i->stream.str().c_str()));
cursor = tmp_cur;
}
pending_streams.clear();
}
Updated by Ernesto Puerta about 5 years ago
After looking at the code of `PyString_FromString`, the issue seems not to be related to the input `c_str()` pointer, but with the `PyObject_Malloc` itself. It's a memory allocator based on malloc for bulk reservations. Python3 has the option to change the memory allocator behaviour via env. variable, but Py2 does not seem to have such a thing. I'll install python-debug in order to display the missing symbols in the stack trace.
Updated by Volker Theile about 5 years ago
I've done some research related to PyObject_Malloc errors and found out that most of the forum comments i found came to the conclusion that mostly an incorrect ref counting is the reason for crashing a Python C extension. The problem is to identify this because the origin of the crash is not where the crash happens.
Updated by Volker Theile about 5 years ago
A good starting point could be https://github.com/ceph/ceph/blob/master/src/mgr/BaseMgrModule.cc#L405 because this is the method that can be found in the log file near every seg fault i came across.
Another one that should have been checked is https://github.com/ceph/ceph/blob/master/src/mgr/ActivePyModules.cc#L529.
Updated by Ernesto Puerta about 5 years ago
- Subject changed from mgr: segmentation fault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc to mgr: segfault, mgr-fin, ActivePyModules::get_python > PyFormatter::get > PyFormatter::finish_pending_streams > PyObject_Malloc
Updated by Sage Weil about 5 years ago
- Status changed from New to Resolved