Actions
Bug #53804
openreproducible segfault in mgr while loading modules on Fedora 35
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I tried to build ceph today and run a vstart cluster, but I'm unable to do so because the mgr keeps crashing.
2022-01-07T11:38:45.077-0500 7f2f1f4c6e40 10 mgr[py] Computed sys.path '/home/jlayton/git/ceph/src/pybind/mgr:/usr/local/lib64/python3.10/site-packages:/usr/local/lib/python3.10/site-packages:/usr/lib64/python3.10/site-packages:/usr/lib/python3.10/site-packages:/home/jlayton/git/ceph/src/pybind:/home/jlayton/git/ceph/build/lib/cython_modules/lib.3:/home/jlayton/git/ceph/src/python-common::/usr/lib64/python310.zip:/usr/lib64/python3.10:/usr/lib64/python3.10/lib-dynload' 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 4 mgr[py] load_subclass_of: found class: 'pg_autoscaler.PgAutoscaler' 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded command osd pool autoscale-status name=format,req=false,positional=false,type=CephString 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded command osd pool set threshold name=num,type=CephFloat 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded command osd pool get noautoscale 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded command osd pool unset noautoscale 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded command osd pool set noautoscale 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 10 mgr[py] loaded 5 commands 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option sleep_interval 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option threshold 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option noautoscale 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option log_level 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option log_to_file 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option log_to_cluster 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 20 mgr[py] loaded module option log_to_cluster_level 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 10 mgr[py] loaded 7 options 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 4 mgr[py] Standby mode not provided by module 'pg_autoscaler' 2022-01-07T11:38:45.134-0500 7f2f1f4c6e40 1 mgr[py] Loading python module 'alerts' 2022-01-07T11:38:45.151-0500 7f2f1f4c6e40 10 mgr[py] Computed sys.path '/home/jlayton/git/ceph/src/pybind/mgr:/usr/local/lib64/python3.10/site-packages:/usr/local/lib/python3.10/site-packages:/usr/lib64/python3.10/site-packages:/usr/lib/python3.10/site-packages:/home/jlayton/git/ceph/src/pybind:/home/jlayton/git/ceph/build/lib/cython_modules/lib.3:/home/jlayton/git/ceph/src/python-common::/usr/lib64/python310.zip:/usr/lib64/python3.10:/usr/lib64/python3.10/lib-dynload' 2022-01-07T11:38:45.182-0500 7f2f1f4c6e40 -1 *** Caught signal (Segmentation fault) ** in thread 7f2f1f4c6e40 thread_name:ceph-mgr ceph version 17.0.0-9986-ge710a41ec0e4 (e710a41ec0e44b52a0184cddab4698e140533a2a) quincy (dev) 1: /lib64/libc.so.6(+0x42750) [0x7f2f1f7a5750] 2: /lib64/libpython3.10.so.1.0(+0x1070e6) [0x7f2f20fa80e6] 3: PyCode_NewWithPosOnlyArgs() 4: /lib64/libpython3.10.so.1.0(+0x106190) [0x7f2f20fa7190] 5: /lib64/libpython3.10.so.1.0(+0x105ddd) [0x7f2f20fa6ddd] 6: /lib64/libpython3.10.so.1.0(+0x106088) [0x7f2f20fa7088] 7: /lib64/libpython3.10.so.1.0(+0x105e33) [0x7f2f20fa6e33] 8: /lib64/libpython3.10.so.1.0(+0x106088) [0x7f2f20fa7088] 9: /lib64/libpython3.10.so.1.0(+0x182c28) [0x7f2f21023c28] 10: /lib64/libpython3.10.so.1.0(+0x195f56) [0x7f2f21036f56] 11: /lib64/libpython3.10.so.1.0(+0x1158d7) [0x7f2f20fb68d7] 12: _PyEval_EvalFrameDefault() 13: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 14: _PyEval_EvalFrameDefault() 15: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 16: _PyEval_EvalFrameDefault() 17: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 18: _PyEval_EvalFrameDefault() 19: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 20: _PyEval_EvalFrameDefault() 21: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 22: _PyEval_EvalFrameDefault() 23: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 24: /lib64/libpython3.10.so.1.0(+0x115a9e) [0x7f2f20fb6a9e] 25: _PyObject_CallMethodIdObjArgs() 26: PyImport_ImportModuleLevelObject() 27: _PyEval_EvalFrameDefault() 28: /lib64/libpython3.10.so.1.0(+0x107984) [0x7f2f20fa8984] 29: PyEval_EvalCode() 30: /lib64/libpython3.10.so.1.0(+0x18a091) [0x7f2f2102b091] 31: /lib64/libpython3.10.so.1.0(+0x1164b0) [0x7f2f20fb74b0] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
I was able to get a core, and the stack trace looks like:
(gdb) bt
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007f2f1f7f28b3 in __pthread_kill_internal (signo=11, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007f2f1f7a56a6 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#3 0x000055ab1f7c394d in reraise_fatal (signum=signum@entry=11) at /home/jlayton/git/ceph/src/global/signal_handler.cc:88
#4 0x000055ab1f7c60fa in handle_oneshot_fatal_signal (signum=11) at /home/jlayton/git/ceph/src/global/signal_handler.cc:363
#5 <signal handler called>
#6 0x00007f2f20fa80e6 in intern_strings (tuple=0x7f2f06d8ee40) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/object.h:146
#7 0x00007f2f20fa7def in PyCode_NewWithPosOnlyArgs (argcount=3, posonlyargcount=<optimized out>, kwonlyargcount=0, nlocals=<optimized out>, stacksize=2, flags=67, code=0x7f2f06d840c0, consts=0x7f2f06d63d80, names=0x7f2f06d84070,
varnames=0x7f2f06d8ee40, freevars=0x7f2f07cb8c40, cellvars=0x7f2f07cb8c40, filename=0x7f2f07966c60, name=0x7f2f06d8ee6f, firstlineno=132, linetable=0x7f2f06d852c0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/codeobject.c:156
#8 0x00007f2f20fa7190 in r_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1378
#9 0x00007f2f20fa6ddd in r_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1174
#10 0x00007f2f20fa7088 in r_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1350
#11 0x00007f2f20fa6e33 in r_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1174
#12 0x00007f2f20fa7088 in r_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1350
#13 0x00007f2f21023c28 in read_object (p=0x7ffebbda4700) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1445
#14 0x00007f2f21036f56 in marshal_loads_impl (bytes=0x7ffebbda4740, bytes=0x7ffebbda4740, module=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/marshal.c:1754
#15 marshal_loads (module=<optimized out>, arg=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/clinic/marshal.c.h:148
#16 0x00007f2f20fb68d7 in cfunction_vectorcall_O (func=0x7f2f0735a0c0, args=0x7f2f07064cb8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/methodobject.c:516
#17 0x00007f2f20faec80 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f07064cb8, callable=0x7f2f0735a0c0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#18 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f07064cb8, callable=0x7f2f0735a0c0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda48a0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#20 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4197
#21 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f07064b20, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#22 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#23 0x00007f2f20faacba in _PyObject_VectorcallTstate (kwnames=0x7f2f073518c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7f2f07060c10, tstate=0x55ab20acd9e0)
at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#24 PyObject_Vectorcall (kwnames=0x7f2f073518c0, nargsf=<optimized out>, args=<optimized out>, callable=0x7f2f07060c10) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#25 call_function (kwnames=0x7f2f073518c0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda4ae0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#26 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4247
#27 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x55ab2195d690, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#28 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#29 0x00007f2f20fa9d6d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06f3cc40, callable=0x7f2f07061750, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#30 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06f3cc40, callable=0x7f2f07061750) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#31 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda4d20, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#32 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4214
#33 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f06f3cac0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#34 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#35 0x00007f2f20fa9d6d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06f2c980, callable=0x7f2f070612d0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#36 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06f2c980, callable=0x7f2f070612d0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#37 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda4f60, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#38 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4214
#39 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f06f2c800, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#40 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#41 0x00007f2f20fa9ac2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06fea9f0, callable=0x7f2f07321d80, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#42 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f06fea9f0, callable=0x7f2f07321d80) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#43 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda51a0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4229
#45 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f06fea840, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#46 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#47 0x00007f2f20fa9ac2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070cba10, callable=0x7f2f07322f80, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#48 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070cba10, callable=0x7f2f07322f80) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#49 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda53e0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#50 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4229
#51 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f070cb880, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#52 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#53 0x00007f2f20fb6a9e in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffebbda55a0, callable=0x7f2f07323010, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#54 object_vacall (tstate=0x55ab20acd9e0, base=<optimized out>, callable=0x7f2f07323010, vargs=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:734
#55 0x00007f2f20fc047c in _PyObject_CallMethodIdObjArgs (obj=0x7f2f0732f6f0, name=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:825
--Type <RET> for more, q to quit, c to continue without paging--c
#56 0x00007f2f20fc01d7 in import_find_and_load (abs_name=0x7f2f06f8ebb0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1521
#57 PyImport_ImportModuleLevelObject (name=0x7f2f06f8ebb0, globals=<optimized out>, locals=<optimized out>, fromlist=0x7f2f070ec8b0, level=0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1622
#58 0x00007f2f20fadc8e in import_name (level=0x7f2f072c8eb0, fromlist=0x7f2f070ec8b0, name=0x7f2f06f8ebb0, f=<optimized out>, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:6021
#59 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:3711
#60 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f07a13880, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#61 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#62 0x00007f2f210240d4 in PyEval_EvalCode (co=0x7f2f06f92a20, globals=0x7f2f06f6f6c0, locals=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:1134
#63 0x00007f2f2102b091 in builtin_exec_impl (module=<optimized out>, locals=0x7f2f06f6f6c0, globals=0x7f2f06f6f6c0, source=0x7f2f06f92a20) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/bltinmodule.c:1056
#64 builtin_exec (module=<optimized out>, args=args@entry=0x7f2f06f8ee18, nargs=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/clinic/bltinmodule.c.h:371
#65 0x00007f2f20fb74b0 in cfunction_vectorcall_FASTCALL (func=0x7f2f07317380, args=0x7f2f06f8ee18, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/methodobject.c:430
#66 0x00007f2f20fb0209 in do_call_core (kwdict=0x7f2f06f6f7c0, callargs=0x7f2f06f8ee00, func=0x7f2f07317380, trace_info=0x7ffebbda5ba0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5920
#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4293
#68 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f0707e050, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#69 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=3, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#70 0x00007f2f20faec80 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f071401c8, callable=0x7f2f073211b0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#71 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f071401c8, callable=0x7f2f073211b0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#72 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda5de0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4197
#74 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f07140040, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#75 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#76 0x00007f2f20fa9d6d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f072b6690, callable=0x7f2f070612d0, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#77 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f072b6690, callable=0x7f2f070612d0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#78 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda6020, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#79 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4214
#80 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f072b6510, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#81 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#82 0x00007f2f20fa9ac2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070969f0, callable=0x7f2f07321d80, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#83 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070969f0, callable=0x7f2f07321d80) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#84 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda6260, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#85 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4229
#86 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f07096840, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#87 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#88 0x00007f2f20fa9ac2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070965d0, callable=0x7f2f07322f80, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#89 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f2f070965d0, callable=0x7f2f07322f80) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:123
#90 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, trace_info=0x7ffebbda64a0, tstate=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5896
#91 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:4229
#92 0x00007f2f20fa8984 in _PyEval_EvalFrame (throwflag=0, f=0x7f2f07096440, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/internal/pycore_ceval.h:46
#93 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=2, kwnames=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/ceval.c:5081
#94 0x00007f2f20fb6a9e in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffebbda6660, callable=0x7f2f07323010, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:114
#95 object_vacall (tstate=0x55ab20acd9e0, base=<optimized out>, callable=0x7f2f07323010, vargs=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:734
#96 0x00007f2f20fc047c in _PyObject_CallMethodIdObjArgs (obj=0x7f2f0732f6f0, name=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:825
#97 0x00007f2f20fc01d7 in import_find_and_load (abs_name=0x7f2f070a6470, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1521
#98 PyImport_ImportModuleLevelObject (name=0x7f2f070a6470, globals=<optimized out>, locals=<optimized out>, fromlist=0x7f2f0732a580, level=0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1622
#99 0x00007f2f20fc746c in builtin___import__ (self=<optimized out>, args=<optimized out>, kwds=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/bltinmodule.c:272
#100 0x00007f2f20fb6e52 in cfunction_call (func=0x7f2f07316f20, args=<optimized out>, kwargs=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/methodobject.c:543
#101 0x00007f2f20fb2ad4 in _PyObject_MakeTpCall (tstate=0x55ab20acd9e0, callable=0x7f2f07316f20, args=0x7ffebbda6910, nargs=5, keywords=0x0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:215
#102 0x00007f2f20fa43a7 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffebbda6910, callable=0x7f2f07316f20, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:112
#103 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7ffebbda6910, callable=0x7f2f07316f20, tstate=0x55ab20acd9e0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/cpython/abstract.h:99
#104 _PyObject_CallFunctionVa (tstate=0x55ab20acd9e0, callable=0x7f2f07316f20, format=<optimized out>, va=<optimized out>, is_size_t=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:485
#105 0x00007f2f20fa4163 in PyObject_CallFunction (callable=<optimized out>, format=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/call.c:507
#106 0x00007f2f20fc7280 in PyImport_Import (module_name=0x7f2f070a6470) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1827
#107 0x00007f2f2102df4d in PyImport_ImportModule (name=<optimized out>) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Python/import.c:1234
#108 0x000055ab1f6562cd in PyModule::load_subclass_of (this=this@entry=0x55ab20acdb10, base_class=base_class@entry=0x55ab1f8133d4 "MgrModule", py_class=py_class@entry=0x55ab20acdc08) at /home/jlayton/git/ceph/src/mgr/PyModule.cc:707
#109 0x000055ab1f657dc0 in PyModule::load (this=this@entry=0x55ab20acdb10, pMainThreadState=<optimized out>) at /home/jlayton/git/ceph/src/mgr/PyModule.cc:378
#110 0x000055ab1f65df5d in PyModuleRegistry::init (this=this@entry=0x7ffebbda98b8) at /usr/include/c++/11/bits/shared_ptr_base.h:1295
#111 0x000055ab1f63973e in MgrStandby::init (this=this@entry=0x7ffebbda71e0) at /home/jlayton/git/ceph/src/mgr/MgrStandby.cc:203
#112 0x000055ab1f50e167 in main (argc=5, argv=0x7ffebbda9c98) at /home/jlayton/git/ceph/src/ceph_mgr.cc:69
Updated by Jeff Layton over 2 years ago
Some more debugging. It looks like the "varnames" variable passed to intern_strings was valid:
Core was generated by `/home/jlayton/git/ceph/build/bin/ceph-mgr -i x -c /home/jlayton/git/ceph/build/'.
Program terminated with signal SIGSEGV, Segmentation fault.
--Type <RET> for more, q to quit, c to continue without paging--c
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
44 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
[Current thread is 1 (Thread 0x7f2f1f4c6e40 (LWP 153587))]
(gdb) f 6
#6 0x00007f2f20fa80e6 in intern_strings (tuple=0x7f2f06d8ee40) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Include/object.h:146
146 return ob->ob_type == type;
(gdb) list
141
142
143 static inline int _Py_IS_TYPE(const PyObject *ob, const PyTypeObject *type) {
144 // bpo-44378: Don't use Py_TYPE() since Py_TYPE() requires a non-const
145 // object.
146 return ob->ob_type == type;
147 }
148 #define Py_IS_TYPE(ob, type) _Py_IS_TYPE(_PyObject_CAST_CONST(ob), type)
149
150
(gdb) p ob
No symbol "ob" in current context.
(gdb) p type
No symbol "type" in current context.
(gdb) f 7
#7 0x00007f2f20fa7def in PyCode_NewWithPosOnlyArgs (argcount=3, posonlyargcount=<optimized out>, kwonlyargcount=0, nlocals=<optimized out>, stacksize=2, flags=67, code=0x7f2f06d840c0,
consts=0x7f2f06d63d80, names=0x7f2f06d84070, varnames=0x7f2f06d8ee40, freevars=0x7f2f07cb8c40, cellvars=0x7f2f07cb8c40, filename=0x7f2f07966c60, name=0x7f2f06d8ee6f,
firstlineno=132, linetable=0x7f2f06d852c0) at /usr/src/debug/python3.10-3.10.1-2.fc35.x86_64/Objects/codeobject.c:156
156 if (intern_strings(varnames) < 0) {
(gdb) p varnames
$1 = (PyObject *) 0x7f2f06d8ee40
(gdb) p *varnames
$2 = {ob_refcnt = 2, ob_type = 0x7f2f211d6620 <PyTuple_Type>}
(gdb)
I don't see the cause of the crash right offhand.
Actions