Project

General

Profile

Actions
Copy link

Bug #15454

closed

bad rados_buffer_free from librados python bindings

Added by Sage Weil about 8 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2016-04-09T21:40:05.606 INFO:tasks.workunit.client.0.smithi002.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:1777: test_mon_ping:  ceph ping 'mon.*'
2016-04-09T21:40:05.908 INFO:tasks.workunit.client.0.smithi002.stderr:*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f27a30115cc ***
2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:======= Backtrace: =========
2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libc.so.6(+0x7b184)[0x7f27a1e3d184]
2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/librados.so.2(rados_buffer_free+0x21)[0x7f27909b4d01]
2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/usr/lib64/python2.7/site-packages/rados.so(+0x28780)[0x7f279a0f0780]
2016-04-09T21:40:05.929 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x50c2)[0x7f27a2b895d2]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4350)[0x7f27a2b88860]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4350)[0x7f27a2b88860]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x7ed)[0x7f27a2b8a0bd]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x6ef68)[0x7f27a2b16f68]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7f27a2af20b3]
2016-04-09T21:40:05.930 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x590a5)[0x7f27a2b010a5]
2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyObject_Call+0x43)[0x7f27a2af20b3]
2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f27a2b83f07]
2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpython2.7.so.1.0(+0x110842)[0x7f27a2bb8842]
2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libpthread.so.0(+0x7dc5)[0x7f27a2893dc5]
2016-04-09T21:40:05.931 INFO:tasks.workunit.client.0.smithi002.stderr:/lib64/libc.so.6(clone+0x6d)[0x7f27a1eb828d]

/a/sage-2016-04-09_18:09:53-rados-master---basic-smithi/118900
Actions
Copy link
 #1

Updated by Kefu Chai about 8 years ago 

2016-04-10 04:40:05.412757 7f2d454a3700  1 -- 172.21.15.2:6789/0 <== client.? 172.21.15.2:0/937778960 1 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x7f2d58a1ea80 con 0x7f2d588aae80
2016-04-10 04:40:05.412796 7f2d454a3700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7f2d586dbc00 MonSession(client.? 172.21.15.2:0/937778960 is open)
2016-04-10 04:40:05.412806 7f2d454a3700 20 mon.a@0(leader) e1  caps
2016-04-10 04:40:05.412814 7f2d454a3700 10 mon.a@0(leader) e1 handle_ping ping magic: 0 v1
2016-04-10 04:40:05.412845 7f2d454a3700 10 mon.a@0(leader).data_health(10) get_health
2016-04-10 04:40:05.413015 7f2d454a3700 10 mon.a@0(leader) e1 handle_ping reply payload len 4322
2016-04-10 04:40:05.413021 7f2d454a3700  1 -- 172.21.15.2:6789/0 --> client.? 172.21.15.2:0/937778960 -- ping magic: 0 v1 -- ?+0 0x7f2d586dac40
2016-04-10 04:40:05.413335 7f2d454a3700 10 mon.a@0(leader) e1 ms_handle_reset 0x7f2d588aae80 172.21.15.2:0/937778960
2016-04-10 04:40:05.413357 7f2d454a3700 10 mon.a@0(leader) e1 reset/close on session client.? 172.21.15.2:0/937778960

mon.a did return a MPing with layload.

Actions
Copy link
 #2

Updated by Sage Weil about 8 years ago

/a/sage-2016-04-15_10:06:03-rados-wip-sage-testing-distro-basic-smithi/131554

Actions
Copy link
 #3

Updated by Sage Weil almost 8 years ago

  • Status changed from 12 to Resolved
Actions
Copy link
 #4

Updated by Loïc Dachary over 7 years ago

http://qa-proxy.ceph.com/teuthology/loic-2016-10-19_21:00:55-rados-jewel-backports-distro-basic-smithi/486295/teuthology.log
http://qa-proxy.ceph.com/teuthology/loic-2016-10-19_06:45:42-rados-jewel-backports-distro-basic-smithi/485086/teuthology.log

2016-10-19T21:23:42.550 INFO:tasks.workunit.client.0.smithi029.stderr:/home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh:1781: test_mon_ping:  ceph ping mon.b
2016-10-19T21:23:42.609 INFO:tasks.workunit.client.0.smithi029.stderr:2016-10-19 21:23:42.609115 7ff49a223700 -1 WARNING: the following dangerous and experimental features are enabled: *
2016-10-19T21:23:42.615 INFO:tasks.workunit.client.0.smithi029.stderr:*** Error in `python': munmap_chunk(): invalid pointer: 0x00007ff4b302b5f4 ***
2016-10-19T21:23:42.627 INFO:tasks.workunit.client.0.smithi029.stderr:======= Backtrace: =========

It however passes 10 times in a row on the same branch http://pulpito.ceph.com/loic-2016-10-20_09:32:24-rados-jewel-backports-distro-basic-smithi/

Actions
Copy link
 #5

Updated by Josh Durgin over 6 years ago

  • Status changed from Resolved to Need More Info
  • Priority changed from Urgent to Normal
Actions
Copy link
 #6

Updated by Nathan Cutler over 6 years ago

Kefu Chai wrote:

[...]

mon.a did return a MPing with payload.

The Python crash is not happening on ceph ping mon.a - it's happening 4 lines down from that, on ceph ping mon.\*

Nevermind, on closer inspection it's happening randomly, sometimes on mon.a, sometimes on mon.b, sometimes on mon.\*

Actions
Copy link
 #7

Updated by Sage Weil almost 3 years ago

  • Status changed from Need More Info to Closed
Actions
Copy link

Also available in: Atom PDF