Bug #18664
ceph_test_msgr SIGABRT in decode_message, SimpleMessenger, can't decode unknown message type 0 MSG_AUTH=17
Status:
New
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.937764 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd6b5c4f00 send m=0x7fbd368f21c0 i=680 2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.937922 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd6b5c7100 send m=0x7fbd291a5d40 i=681 2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938034 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29545d00 send m=0x7fbd291a52c0 i=682 2017-01-24T20:43:20.389 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938053 7fbd27a55f00 -1 ceph_test_msgr Op 1500: 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938057 7fbd27a55f00 -1 ceph_test_msgr available_connections: 15 inflight messages: 12 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.938155 7fbcb4e4a700 -1 ceph_test_msgr ms_fast_dispatch conn=0x7fbd6b5c7d00reply=^A i = 666 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939210 7fbcb3e3a700 0 -- 127.0.0.1:16814/12764 >> 127.0.0.1:16838/12788 pipe(0x7fbd6119e000 sd=172 :16814 s=0 pgs=0 cs=0 l=0 c=0x7fbd6b5c7800).accept connect_seq 0 vs existing 0 state connecting 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939573 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29545200 send m=0x7fbd291a5100 i=683 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939610 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29542d00 send m=0x7fbd291a5800 i=684 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.939628 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29542a00 send m=0x7fbd291a5640 i=685 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940875 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29542200 send m=0x7fbd291a4f40 i=686 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940898 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29545d00 send m=0x7fbd291a4a00 i=687 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.940975 7fbcb7c78700 0 can't decode unknown message type 0 MSG_AUTH=17 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941004 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd29543d00 send m=0x7fbd291a4140 i=688 2017-01-24T20:43:20.390 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941096 7fbd27a55f00 -1 ceph_test_msgr Op 1510: 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941099 7fbd27a55f00 -1 ceph_test_msgr available_connections: 14 inflight messages: 13 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941109 7fbd27a55f00 -1 ceph_test_msgr send_message_wrap conn=0x7fbd6b5c4f00 send m=0x7fbd291a28c0 i=689 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr:2017-01-24 20:43:18.941949 7fbcb7c78700 -1 *** Caught signal (Aborted) ** 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: in thread 7fbcb7c78700 thread_name:ms_pipe_read 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: ceph version 11.1.0-6995-gf750cad (f750cadc8d0d0816a64e48339f9bd2dce7b2034a) 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 1: (()+0x5ab9f) [0x7fbd27acfb9f] 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 2: (()+0xf370) [0x7fbd27204370] 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 3: (gsignal()+0x37) [0x7fbd1b80b1d7] 2017-01-24T20:43:20.391 INFO:teuthology.orchestra.run.smithi154.stderr: 4: (abort()+0x148) [0x7fbd1b80c8c8] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 5: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&)+0x18a7) [0x7fbd1dcfd157] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 6: (Pipe::read_message(Message**, AuthSessionHandler*)+0x1441) [0x7fbd1dd6a4e1] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 7: (Pipe::reader()+0xbb1) [0x7fbd1dd76dd1] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 8: (Pipe::Reader::entry()+0xd) [0x7fbd1dd80c6d] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 9: (()+0x7dc5) [0x7fbd271fcdc5] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: 10: (clone()+0x6d) [0x7fbd1b8cd73d] 2017-01-24T20:43:20.392 INFO:teuthology.orchestra.run.smithi154.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
/a/sage-2017-01-24_19:52:02-rados-wip-sage-testing---basic-smithi/743886
Related issues
History
#1 Updated by Sage Weil about 7 years ago
- Subject changed from ceph_test_msgr SIGABRT in decode_message, can't decode unknown message type 0 MSG_AUTH=17 to ceph_test_msgr SIGABRT in decode_message, SimpleMessenger, can't decode unknown message type 0 MSG_AUTH=17
#2 Updated by Greg Farnum almost 7 years ago
- Priority changed from Urgent to Normal
Apparently this hasn't reproduced?
#3 Updated by Greg Farnum almost 7 years ago
- Duplicated by Bug #20361: ceph_test_msgr: can't decode unknown message type 0 MSG_AUTH=17 added
#4 Updated by Greg Farnum almost 7 years ago
- Priority changed from Normal to High
/a/sage-2017-06-20_17:41:31-rados-wip-sage-testing-distro-basic-smithi/1306218
#5 Updated by Sage Weil over 6 years ago
- Status changed from New to Can't reproduce
#6 Updated by Sage Weil over 6 years ago
- Status changed from Can't reproduce to 12
/a/sage-2017-09-27_21:32:47-rados-wip-sage-testing-2017-09-27-1558-distro-basic-smithi/1680113
#7 Updated by Sage Weil about 6 years ago
/a/sage-2018-01-09_00:16:55-rados-wip-sage-testing-2018-01-08-1631-distro-basic-smithi/2044757
#8 Updated by Neha Ojha over 5 years ago
Similar failure seen in luminous:
2018-10-23T16:55:17.223 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.224198 7f14439c83c0 -1 ceph_test_msgr send_message_wrap conn=0x55772152fc20 send m=0x557750c74f00 i=889 2018-10-23T16:55:17.224 INFO:teuthology.orchestra.run.smithi193.stderr:*** Caught signal (Aborted) ** 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr: in thread 7f13d0321700 thread_name:ms_pipe_read 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225272 7f13d0321700 0 can't decode unknown message type 0 MSG_AUTH=17 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225315 7f13d405e700 0 -- 127.0.0.1:16833/11971 >> 127.0.0.1:16801/11939 pipe(0x557721db0800 sd=193 :16833 s=2 pgs=15 cs=1 l=0 c=0x55772152e7e0).fault with nothing to send, going to standby 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225434 7f14439c83c0 -1 ceph_test_msgr Op 1810: 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225443 7f14439c83c0 -1 ceph_test_msgr available_connections: 16 inflight messages: 15 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225497 7f14439c83c0 -1 ceph_test_msgr send_message_wrap conn=0x557732f0bc20 send m=0x557750c76760 i=890 2018-10-23T16:55:17.225 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.225712 7f13d203e700 -1 ceph_test_msgr ms_fast_dispatch conn=0x55773f507560reply= 2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: ceph version 12.2.8-504-g943c4b5 (943c4b5a87c3f117458b5b6297beadde2a785ccf) luminous (stable) 2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 1: (()+0x58a54) [0x55771ecdfa54] 2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 2: (()+0x11390) [0x7f14435b4390] 2018-10-23T16:55:17.243 INFO:teuthology.orchestra.run.smithi193.stderr: 3: (gsignal()+0x38) [0x7f1439ade428] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 4: (abort()+0x16a) [0x7f1439ae002a] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 5: (decode_message(CephContext*, int, ceph_msg_header&, ceph_msg_footer&, ceph::buffer::list&, ceph::buffer::list&, ceph::buffer::list&, Connection*)+0x27d1) [0x7f143a922bf1] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 6: (Pipe::read_message(Message**, AuthSessionHandler*)+0x187c) [0x7f143a9f23cc] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 7: (Pipe::reader()+0x1170) [0x7f143a9fea50] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 8: (Pipe::Reader::entry()+0xd) [0x7f143aa08fed] 2018-10-23T16:55:17.244 INFO:teuthology.orchestra.run.smithi193.stderr: 9: (()+0x76ba) [0x7f14435aa6ba] 2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr: 10: (clone()+0x6d) [0x7f1439bb041d] 2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr:2018-10-23 16:55:17.226173 7f13d0321700 -1 *** Caught signal (Aborted) ** 2018-10-23T16:55:17.245 INFO:teuthology.orchestra.run.smithi193.stderr: in thread 7f13d0321700 thread_name:ms_pipe_read
/a/yuriw-2018-10-23_15:16:03-rados-wip-yuri-testing-2018-10-22-2055-luminous-distro-basic-smithi/3175376/
#9 Updated by Neha Ojha over 4 years ago
- Project changed from Ceph to Messengers
/a/nojha-2019-08-26_20:27:46-rados-wip-bluefs-shared-alloc-luminous-2019-08-26-distro-basic-smithi/4255419/
#10 Updated by Patrick Donnelly over 4 years ago
- Status changed from 12 to New