Bug #21593: segv in PyList_New from PyFormatter - mgr - Ceph

Actions

Copy link

Bug #21593

closed

segv in PyList_New from PyFormatter

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

John Spray

Category:

python interface

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Thread 1 (Thread 0x7fd398434700 (LWP 10047)):
#0  0x00007fd40403b16b in raise () from /lib64/libpthread.so.0
#1  0x00005568483f0eb2 in reraise_fatal (signum=11)
    at /home/sage/src/ceph6/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11) at /home/sage/src/ceph6/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  0x00007fd40598dee4 in visit_decref () from /lib64/libpython2.7.so.1.0
#5  0x00007fd4059a4f31 in instancemethod_traverse.lto_priv () from /lib64/libpython2.7.so.1.0
#6  0x00007fd405996db5 in collect.lto_priv () from /lib64/libpython2.7.so.1.0
#7  0x00007fd405967e1c in _PyObject_GC_Malloc () from /lib64/libpython2.7.so.1.0
#8  0x00007fd405967e9d in _PyObject_GC_New () from /lib64/libpython2.7.so.1.0
#9  0x00007fd4059747fc in PyList_New () from /lib64/libpython2.7.so.1.0
#10 0x00005568482d5bf1 in PyFormatter::open_array_section (this=0x7fd398430810, 
    name=0x556848867eef "pg_upmap_items") at /home/sage/src/ceph6/src/mgr/PyFormatter.cc:25
#11 0x000055684851bf7c in OSDMap::dump (this=this@entry=0x7fd390487680, 
    f=f@entry=0x7fd398430810) at /home/sage/src/ceph6/src/osd/OSDMap.cc:2854
#12 0x00005568482d6b0a in osdmap_dump (self=<optimized out>, obj=<optimized out>)
    at /home/sage/src/ceph6/src/mgr/PyOSDMap.cc:36
#13 0x00007fd405a44772 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fd4059861ce in function_call.lto_priv () from /lib64/libpython2.7.so.1.0
#17 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#18 0x00007fd4059a8e7e in instancemethod_call.lto_priv () from /lib64/libpython2.7.so.1.0
#19 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#20 0x00007fd405a3be97 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#21 0x00007fd40599ab2c in PyInstance_New () from /lib64/libpython2.7.so.1.0
#22 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#23 0x00007fd405a40884 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#24 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#25 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#26 0x00007fd405a42863 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#27 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#28 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#29 0x00007fd4059861ce in function_call.lto_priv () from /lib64/libpython2.7.so.1.0
#30 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#31 0x00007fd4059a8e7e in instancemethod_call.lto_priv () from /lib64/libpython2.7.so.1.0
#32 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#33 0x00007fd405960dde in PyObject_CallMethod () from /lib64/libpython2.7.so.1.0
#34 0x00005568482e4592 in MgrPyModule::serve (this=0x7fd3b4014ed0)
    at /home/sage/src/ceph6/src/mgr/MgrPyModule.cc:212

Actions

Copy link

Updated by Sage Weil over 6 years ago

I'm triggering this with github.com/liewegas/ceph wip-balancer,

bin/init-ceph stop ; MON=1 OSD=8 MDS=0 ../src/vstart.sh  -d -n -x -l ; bin/ceph osd pool create foo 128 ; bin/ceph config-key set mgr/balancer/sleep_interval 15 ; bin/ceph mgr module enable balancer ; bin/ceph balancer mode crush-compat ; bin/ceph balancer on ; bin/init-ceph restart 
sleep 10
bin/ceph osd out 2
...

may need to mark in or out a few times? not certain.

in another terminal i'm running

watch -n .1 'bin/ceph osd df ; bin/ceph osd crush tree'

although that is presumably unrelated.

Actions

Copy link

Updated by John Spray over 6 years ago

Category set to python interface
Status changed from 12 to In Progress
Assignee set to John Spray

Actions

Copy link

Updated by John Spray over 6 years ago

Spun off https://github.com/ceph/ceph/pull/18093 while trying to get to bottom of this

Actions

Copy link

Updated by John Spray over 6 years ago

This has obstinately refused to reproduce on my local vstart system, so I've built a branch with the new assertions and re-running against rados/thrash failures here: http://pulpito.ceph.com/jspray-2017-10-04_12:55:50-rados:thrash-wip-21593-debug-distro-basic-smithi/

Actions

Copy link