Project

General

Profile

Bug #21593

segv in PyList_New from PyFormatter

Added by Sage Weil 3 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
python interface
Target version:
-
Start date:
09/28/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

Thread 1 (Thread 0x7fd398434700 (LWP 10047)):
#0  0x00007fd40403b16b in raise () from /lib64/libpthread.so.0
#1  0x00005568483f0eb2 in reraise_fatal (signum=11)
    at /home/sage/src/ceph6/src/global/signal_handler.cc:74
#2  handle_fatal_signal (signum=11) at /home/sage/src/ceph6/src/global/signal_handler.cc:138
#3  <signal handler called>
#4  0x00007fd40598dee4 in visit_decref () from /lib64/libpython2.7.so.1.0
#5  0x00007fd4059a4f31 in instancemethod_traverse.lto_priv () from /lib64/libpython2.7.so.1.0
#6  0x00007fd405996db5 in collect.lto_priv () from /lib64/libpython2.7.so.1.0
#7  0x00007fd405967e1c in _PyObject_GC_Malloc () from /lib64/libpython2.7.so.1.0
#8  0x00007fd405967e9d in _PyObject_GC_New () from /lib64/libpython2.7.so.1.0
#9  0x00007fd4059747fc in PyList_New () from /lib64/libpython2.7.so.1.0
#10 0x00005568482d5bf1 in PyFormatter::open_array_section (this=0x7fd398430810, 
    name=0x556848867eef "pg_upmap_items") at /home/sage/src/ceph6/src/mgr/PyFormatter.cc:25
#11 0x000055684851bf7c in OSDMap::dump (this=this@entry=0x7fd390487680, 
    f=f@entry=0x7fd398430810) at /home/sage/src/ceph6/src/osd/OSDMap.cc:2854
#12 0x00005568482d6b0a in osdmap_dump (self=<optimized out>, obj=<optimized out>)
    at /home/sage/src/ceph6/src/mgr/PyOSDMap.cc:36
#13 0x00007fd405a44772 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#14 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#15 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#16 0x00007fd4059861ce in function_call.lto_priv () from /lib64/libpython2.7.so.1.0
#17 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#18 0x00007fd4059a8e7e in instancemethod_call.lto_priv () from /lib64/libpython2.7.so.1.0
#19 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#20 0x00007fd405a3be97 in PyEval_CallObjectWithKeywords () from /lib64/libpython2.7.so.1.0
#21 0x00007fd40599ab2c in PyInstance_New () from /lib64/libpython2.7.so.1.0
#22 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#23 0x00007fd405a40884 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#24 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#25 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#26 0x00007fd405a42863 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#27 0x00007fd405a431c9 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#28 0x00007fd405a45828 in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#29 0x00007fd4059861ce in function_call.lto_priv () from /lib64/libpython2.7.so.1.0
#30 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#31 0x00007fd4059a8e7e in instancemethod_call.lto_priv () from /lib64/libpython2.7.so.1.0
#32 0x00007fd40595dea3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#33 0x00007fd405960dde in PyObject_CallMethod () from /lib64/libpython2.7.so.1.0
#34 0x00005568482e4592 in MgrPyModule::serve (this=0x7fd3b4014ed0)
    at /home/sage/src/ceph6/src/mgr/MgrPyModule.cc:212

History

#1 Updated by Sage Weil 3 months ago

I'm triggering this with github.com/liewegas/ceph wip-balancer,

bin/init-ceph stop ; MON=1 OSD=8 MDS=0 ../src/vstart.sh  -d -n -x -l ; bin/ceph osd pool create foo 128 ; bin/ceph config-key set mgr/balancer/sleep_interval 15 ; bin/ceph mgr module enable balancer ; bin/ceph balancer mode crush-compat ; bin/ceph balancer on ; bin/init-ceph restart 
sleep 10
bin/ceph osd out 2
...

may need to mark in or out a few times? not certain.

in another terminal i'm running

watch -n .1 'bin/ceph osd df ; bin/ceph osd crush tree'

although that is presumably unrelated.

#2 Updated by John Spray 3 months ago

  • Category set to python interface
  • Status changed from Verified to In Progress
  • Assignee set to John Spray

#3 Updated by John Spray 2 months ago

Spun off https://github.com/ceph/ceph/pull/18093 while trying to get to bottom of this

#4 Updated by John Spray 2 months ago

This has obstinately refused to reproduce on my local vstart system, so I've built a branch with the new assertions and re-running against rados/thrash failures here: http://pulpito.ceph.com/jspray-2017-10-04_12:55:50-rados:thrash-wip-21593-debug-distro-basic-smithi/

#5 Updated by John Spray about 2 months ago

  • Status changed from In Progress to Need Review

#6 Updated by John Spray about 1 month ago

  • Status changed from Need Review to Resolved

NB this will also show up in Luminous via https://github.com/ceph/ceph/pull/18675

Also available in: Atom PDF