Bug #15454
closedbad rados_buffer_free from librados python bindings
0%
Description
2016-04-09T21:40:05.606 INFO:tasks.workunit.client.0.smithi002.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:1777: test_mon_ping: ceph ping 'mon.*' 2016-04-09T21:40:05.908 INFO:tasks.workunit.client.0.smithi002.stderr:*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f27a30115cc *** 2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:======= Backtrace: ========= 2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libc.so.6(+0x7b184)[0x7f27a1e3d184] 2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/librados.so.2(rados_buffer_free+0x21)[0x7f27909b4d01] 2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/usr/lib64/python2.7/site-packages/rados.so(+0x28780)[0x7f279a0f0780] 2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x50c2)[0x7f27a2b895d2] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4350)[0x7f27a2b88860] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4350)[0x7f27a2b88860] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f27a2b8a0bd] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x6ef68)[0x7f27a2b16f68] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7f27a2af20b3] 2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x590a5)[0x7f27a2b010a5] 2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7f27a2af20b3] 2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f27a2b83f07] 2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x110842)[0x7f27a2bb8842] 2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpthread.so.0(+0x7dc5)[0x7f27a2893dc5] 2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libc.so.6(clone+0x6d)[0x7f27a1eb828d]
/a/sage-2016-04-09_18:09:53-rados-master---basic-smithi/118900
Updated by Kefu Chai about 8 years ago
2016-04-10 04:40:05.412757 7f2d454a3700 1 -- 172.21.15.2:6789/0 <== client.? 172.21.15.2:0/937778960 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x7f2d58a1ea80 con 0x7f2d588aae80 2016-04-10 04:40:05.412796 7f2d454a3700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7f2d586dbc00 MonSession(client.? 172.21.15.2:0/937778960 is open) 2016-04-10 04:40:05.412806 7f2d454a3700 20 mon.a@0(leader) e1 caps 2016-04-10 04:40:05.412814 7f2d454a3700 10 mon.a@0(leader) e1 handle_ping ping magic: 0 v1 2016-04-10 04:40:05.412845 7f2d454a3700 10 mon.a@0(leader).data_health(10) get_health 2016-04-10 04:40:05.413015 7f2d454a3700 10 mon.a@0(leader) e1 handle_ping reply payload len 4322 2016-04-10 04:40:05.413021 7f2d454a3700 1 -- 172.21.15.2:6789/0 --> client.? 172.21.15.2:0/937778960 -- ping magic: 0 v1 -- ?+0 0x7f2d586dac40 2016-04-10 04:40:05.413335 7f2d454a3700 10 mon.a@0(leader) e1 ms_handle_reset 0x7f2d588aae80 172.21.15.2:0/937778960 2016-04-10 04:40:05.413357 7f2d454a3700 10 mon.a@0(leader) e1 reset/close on session client.? 172.21.15.2:0/937778960
mon.a did return a MPing with layload.
Updated by Sage Weil about 8 years ago
/a/sage-2016-04-15_10:06:03-rados-wip-sage-testing-distro-basic-smithi/131554
Updated by Loïc Dachary over 7 years ago
http://qa-proxy.ceph.com/teuthology/loic-2016-10-19_21:00:55-rados-jewel-backports-distro-basic-smithi/486295/teuthology.log
http://qa-proxy.ceph.com/teuthology/loic-2016-10-19_06:45:42-rados-jewel-backports-distro-basic-smithi/485086/teuthology.log
2016-10-19T21:23:42.550 INFO:tasks.workunit.client.0.smithi029.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:1781: test_mon_ping: ceph ping mon.b 2016-10-19T21:23:42.609 INFO:tasks.workunit.client.0.smithi029.stderr:2016-10-19 21:23:42.609115 7ff49a223700 -1 WARNING: the following dangerous and experimental features are enabled: * 2016-10-19T21:23:42.615 INFO:tasks.workunit.client.0.smithi029.stderr:*** Error in `python': munmap_chunk(): invalid pointer: 0x00007ff4b302b5f4 *** 2016-10-19T21:23:42.627 INFO:tasks.workunit.client.0.smithi029.stderr:======= Backtrace: =========
It however passes 10 times in a row on the same branch http://pulpito.ceph.com/loic-2016-10-20_09:32:24-rados-jewel-backports-distro-basic-smithi/
Updated by Josh Durgin over 6 years ago
- Status changed from Resolved to Need More Info
- Priority changed from Urgent to Normal
Happened in 10.2.10 validation a couple times:
http://pulpito.ceph.com/yuriw-2017-09-20_15:28:55-rados-jewel-distro-basic-smithi/1652571/
http://pulpito.ceph.com/yuriw-2017-09-19_20:43:32-rados-jewel-distro-basic-smithi/1649152/
Updated by Nathan Cutler over 6 years ago
Kefu Chai wrote:
[...]
mon.a did return a MPing with payload.
The Python crash is not happening on ceph ping mon.a
- it's happening 4 lines down from that, on ceph ping mon.\*
Nevermind, on closer inspection it's happening randomly, sometimes on mon.a, sometimes on mon.b, sometimes on mon.\*
Updated by Sage Weil almost 3 years ago
- Status changed from Need More Info to Closed